ERIC 



DOCUMENT RESUME 



ED 243 246 



EA 016 742 



AUTHOR 
TITLE 



INSTITUTION 

SPONS AGENCY 
PUB DATE 
GRANT 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



Williams, Richard C. ; Bank, Adrianne 
Improving Instruction through the Management of 
Testing and Evaluation Activities: A Guidebook for 
School Districts. 

California Univ., Los Angeles .'Center for the Study 
of Evaluation . 

National Inst, of Education (ED), Washington, DC. 
Nov 82 

NIE-G-80-0112,P5 s 

299p.; Evaluation Design Project. Supersedes ED 226 

016. 

Guides - Non-Classroom Use (055) 
MF01/PC12 Plus Postage. 

Academic Achievement; Case Studies; *Change 
Strategies; Curriculum Evaluation; *Educational 
Planning; Educational Strategies; Elementary 
Secondary Education; *Evaluation Utilization; \' 
Instructional Development; ^Instructional 
Improvement; *Pro'gram Evaluation'; *Test. 
Interpretation; Test Use 



ABSTRACT 

This guidebook shows how school district personnel 
from" many areas—curriculum, instruction, supervision, staff 
development — tan coordinate their activities to focus on * / 
instructional improvement. It reveals how data derived from tests, 
when properly used, can be useful in helping district personnel work 
with school and community people to assess the adequacy of an 
instructional program and to improve it. After an introductory 
overview of the guide, chapter 2 presents a scenario of a week in a / 
relatively small school district where declining, test .scores have 
just been released to the newspapers . The ongoing activities, daily 
crises, and events of the school day form a backdrop against which 
board members and administrators attempt to assess the significance 
of the scores and to *plan their act ions accordingly . Chapter 3 
addresses assumptions underlying questions or demands for action from 
the public, in response to release of test scores. These "assumptions 
wrongly assume that a single factor is responsible for low • 
achievement, but actually no single procedure, material, facility, or 
person operates in isolation from the complex**interactions in an 
educational system. In chapter 4 f six districts tell how they 
responded in a unique way to the problem of increasing student 
achievement. Each used test scores and evaluative data not only to 
monitor but to plan instruction. Chapter 5 identifies five reasons 
why a long-term coordinated district strategy is difficult to 
achieve. Ail open systems perspective then shows how changed 
assumptions can direct a school district toward a coherent plan to 
. interpret testing, evaluation, and instruction. Chapter 6 provides 
(1) reasons for developing a uhique evaluation system, (2) starting 
points (opportunities and constraints), (3) a sequence for 
development, (4) selection of emphasis, and (5) methods for 
identifying sources of support* (TE) 
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PREFACE ' - , 

* 

. From the beginning of formal education, parents and pupils and teachers 
have been interested' in, and concerned about what and how children learn. 

' In the past, when pupils did not perform at what adults regarded as 
acceptable levels the blame was often placed on the pupils 1 lack of ability 
or unwillingness to apply him or herself to the academic tasks that were set 
out. Of course, teachers* received a share of the blame as well. 

More recently, notice has been, taken of the role that parent and. peer 
attitudes may play in influencing students to value, and to prosper in an 

educational system. 

Even so; in addressing the issue of how to increase learning, the focus 
of analysis^ has- been on the urlique interchange between teacher and pupil as 
it takes place within the schox>l classroom. 

In the' last 20 years, federal and state governments have begun funding 
specific programs intended to ameliorate what some have seen as persistent 
problems of the^chooling process. Riding aloncf wi.th this external funding 
has come a new focus for educational analysis, Namely the extent to which 
pupil achievement is, influenced by educational progranfe and by school dis- 

L 

trict level . actions. 

^ This shift in level of analysis has been accompanied by a new set of 
questions addled to concern with pupil .motivation, teacher behavior, peer 
interactions, arjd* parent -encouragement. These concerns yith program deli- 
very procjesses^ adequacy of implementation assessed by administrative moni- 
toring, and testing qf student achievement, which had historically been a 
measuring tool so that teachers could determine pupil achievement, became 
folded into the newer field, of program evaluation. 



Program evaluation used tests to assess pupil programs and other evalu- 
ative techniques -.- classroom visitation, surveys, opinionnai res, to' de- 
scribe program inputs. Ideally, data about the educational messages deli- 
vered to students and student learning from those messages should lead 
school personnel to a determination of where action should be taken to 
enhance student learning. 

This book is our attempt to analyze the bridges that a district can ' . 
build between test and evaluation data on the one hand, and school district 
instructional decision making on the other. Our analysis is not focused on 
the teacher in the classroom, nor on the principal in the school building.' 
Nerther is it )n the administrator of the special program. Rather, it is on 
the personnel in the school district's central office. We have selected 
this'level because we believe many, but certainly not all, pupil achievement 
problems can be and, indeed, must be, adjressed at that level. Given the 
way our public school system is presently structured, the ultimate opera- 
tional responsibility for pupil achievement must rest at the district 
1 eve.l . 

'This book, focusing on the district level, represents a relatively new 
approach to this prgblem of improving pupil achievement. 

We suppose that if there were any word that would typify the book's 
spirit and structure, it would be the word synthesi s . This synthesis is re- 
presented in several ways: the subject matter, the operational level, and ■ 
the contributors. To elaborate: First, the subject matter represents a 
synthesis of several fields of inquiry^ We attend to the characteristics of 
and interrelationships between testing, evaluation and instruction on the 
one hand, and organizational theory on the other. Second, at the* 



operational level, we try to integrate theory and practice, or put another 
way, analysis with action* There is ho formula or pre-f abricated program 
that will guarantee a school district an effective way to link testing, and 
evaluation with instruction to improve pupil achievement* Developing such i 
program depends upon analyzing each district's unique problems, make-up and 
setting and devising a program to meet local real ities. Third, the book's 
contributors represent experiences gained through detailed observation and 
academic searchings coupled with experiences gained through daily struggle 
with designing, developing, and implementing programs for linking testing 
and evaluation with instruction in school districts. 

Blending and synthesizing ideas generated from. diverse experiences has 

c . 

been sometimes difficult, but always, stimulating. While our own knowledge 
and interest in this topic has grown dramatical ly • in the process of conduct 
ing our research and working with a wide array of individuals, the ultimate 
success -of the book will rest on whether or not our work helps school 
districts to better develop and implement more effective instructional 
programs* 



r 



CHAPTER 1 



•What' This Book is About ' ' 

in which the major, perspectives of the book are 
summarized along with a precis for each, of the 
remaining chapters ... 4 
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CHAPTER 1 * 



What This Book is About 



This book is about how school districts can help their students learn 



This book says that school district personnel who work in many areas — 
curriculum, instruction, supervision, staff development — tan* coordinate 
their activities so as to focus on instructional improvement. 

This book says that data derived from tests, when properly used, ana- 
lyzed, interpreted j can be useful in helping district personnel in many. 



areas work with school and community people to assess the adequacy of the 
instructional program and to improve it. 

Many districts, at present, collect a lot of^4ata. Much of it remains 



merely data, undigested, unused, or worse* transformed into misinformation. 
Low test scores, when reported by the media, and read by the public may 



the inadequacy of student achievement. The problem, then, is framed by the 
question, "How to we raise test scores? 1 ' 

Chapter to presents a scenario of a week in a relatively small 
school district where test scores have just been released to the news- 
papers. The ongoing activities, daily crises, and. events, the persona- 
lities of individuals, formthe backdrop against which board members, 
the superintendent, administrators and teachers ask themselves and one 
another - "What dp the scores mean?" "What next? 11 "Where do we go 
from here?" ' 



more. 




appear to be an accurate assessment of the lack of quality of schools and 
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Education has the unique distinction of being a topic that almost every 
segment of society feels equipped to judge and express convictions.. These 
feelings and convictions stem from the fact that theJ\merican system of 
public edi*catioxi has offered an "educational program for all youth, and from 
the personal experience individuals tend to draw retrospective conclusions 
regarding the relation of the educational experience to successes, failures, 
wishes and unsolved problems both during school and in the post-school 
years. 

A common reaction to the many problems of society is to look for a 
cause or reason for the undesired circumstances. In the search for a 'single 
caus-e or reason for complex problems, it is common for a single content, 
process* or procedure to be identified as the "cause" and to suggest that a 
change will eliminate the problem. j 

The organization, administration and practices of education are under 
close scrutiny by. all r segments of society. The questions raised about edu- 
cational practices are often not asked to obtain explanation or information, 
but are statements of belief regarding a solution to a perceived problem. ^ 
For example, questions-Such as — "How soon, can we expect test scores to go 
up?"" "Why don't weidentify incompetent teachers and do something about 
them?" "Which district has an instructional program that has been proven/to 
improve test scores?" and "Why donJt we ask the Superintendent to develop a 
comprehensive plan to improve student achievement and present the plan and 
the related costs next month?" — are stated as questions but are also . 
statements of belief of what should.be done to improve the educational pro- 
gram and eliminate the condition that hls~\eated the perceived prcblem. 
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- Chapter 3 takes each of these questions and explores why the 
question is misleading. 

These questions or demands for action are based upon the erroneous 
assumption that a single factor is responsible for low achievement, 
poor discipline, low morale or whatever. The statement of the question 
, commonly assumes that a quick and comprehensive change will be observed 
from the change of a single element, a single pronouncement by the 
admini stration, or the substitution of new materials or techniques. 

This^ book suggests an alternative to the quick fix. We suggest that no 
single procedure, material, facility or person operates in isolation from 
the complex interactions among ongoing operations in the educational 
system. 

In Chapter 4, six districts tell their stories* Each of these 
district's responded to the problem of increasing student achievement in 
a unique way. Each district, starting from a sense of district respon- 
sibility. for student learning, evolved, over a'period of ti^, a method 
whereby the district used test scores and evaluative data not only to 
monitor but also to plan instruction; not only to diagnose but also to 
prescribe. Two of these districts developed centralized curricula- 
based systems where tests assisted teachers in increasing children's 
learning through a prescribed, sequent of knowledge acquisition and 
skill development. Two others developed a feedback method, where 
central office staff interpreted test and evaluative data to school 
planning teams who advised on annual resource allocations. 

Particular districts who approach the student achievement question in 

integrated organizational terms have developed such systems. But there are 

not many whfr do so. Why are these districts different -from others? Or, to 

turn the v question around, whatare«the impediments to all districts in doing 

■A. 4 

the same thing? Why is instructional improvement .through the management of 

e 

testing and evaluation not a natural ly. occurring phenomenon.- 

Chapter 5, using the scenario and district stories', points out < 
- five reasons wiSy a long-term coordinated district strategy is difficult 
to conceptualize. However, an open-systems perspective explains some ' 
of the barriers. It provides an understanding of how these changed 
assumptions cati lead a School district in the direction of a coherent 
plan to interpret testing, evaluation and instruction. , 



Chapter 6 will personalize the foregoing discussions so that dis- ' 
tricts may begin to think about: 1) reasons for developing a unique 
system; 2) starting points, including analyzing opportunities and cons- 
traints posed by the environment and history; identi fyi ng" existi ng , 
ideas about testing, evaluation., and instruction; and looking a\ cur- 
rent curriculum, instruction and testing activities and what can be 
changed; 3) sequencing for development; 4) selecting emphasis (e.g., 
staff development, curriculum and materials, tests and evaluations); 5) 
identifying sources of support — external, administrative and 
technical. " ' 
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CHAPTER 2 t 



School District Scenario 

in which district perspnrfel consider public 
reaction to test score decline as freported 
in the media . . . * 



CHAPTER 2 
School District Scenario 

It's Monday, well after 7:30 a.m. The district office of the _ j 

Unified School District is al ready ful l~staf fed with personnel performing 
their designated functions. A new bus route just has been created for those 
students attending a self-contained class for the Educationally Handi- 
capped. The Director of Food Services is dispatching the Monday menu of. 
spaghetti, mixed vegetables, jello and milk to all of the district's kit- 
chens. The clerk in charge of contacting teacher substitutes has made the 
usual wake-up calls and now operates the switchboard, third cup of coffee in 
hand. The Business Services branch hurriedly types a contractual agreement 

r - 

for today's staff development presentation. 

The district is moderate in size with a diverse student populaticm. 
Minority enrollment is on the increase, yet the total student enrollment is 
on the decline. There are fourteen elementary schools, two middle and two 
senior high schools. The staff has been stable over the years, but within 
the past two there have been many between-school teacher transfer requests^ 
The teachers' organization endorses the transfers and is pushing hard to 
lower the administrator-teacher ratio and raise teacher salaries by 15 
.percent. * 

-o A few parents have already started calling the Superintendent's Office 
regarding last evening's newspaper article headlined -- "Secondary Scores 
Decline--Elementary Tests Remain Stable." Some of these callers are 



concerned about what the published percentiles mean for their children's 
chances to get into good colleges. Two citizens who had participated on the 
district's committee developing system-wide goals and instructional objec- 
tives, find it hard to understand that the district students haven't yet 
made any gains in. reading and mathematics. One taxpayer suggests that 
dwindling federal dollars are being squandered through bureaucratic wrong- 

doinqs. One call is taken by the curriculum specialist who attempts to ex- 

\ 

plain the scores in terms of changing student populations ancTthe shifting 
district emphasis into bilingual and career education programs. A final 
phone conversation focuses on a related news article about the upcoming 
local tax election and the political involvements of some voc: 1 Board 
member. 

The Superintendent, Chris Dewey, faces another crowded day's agenda. 
The 'day-at-a-glance calendar sprawl ed on the desk indicates that the 1 regular 
scheduled monthly meeting, with his principals begins at nine o'clock. Pen- 
cilled i-n red for 8:45 this morning is a hastily called meeting with two 
school Board members who want some quick answers about the test score situ- 
ation. At one o'clock, Chris wilTmeet with' the Executive Cabinet, which v 
consists of the district's central oTTTce administrators. 

On one elementary campus, some of the teachers convene in the lounge 
and discuss the newspaper story. A group of instructional aides hired 
through Title I funds are there too. Designated staff members are perform- 
ing their normal supervision duties on the playground. The students seem 
happy. The majority of students are clustered in their own ethnic groups, 
moving about with lunches, instruments and backpacks.^ They play and chatter 
informally until the morning bell calls them inside to classrooms. 
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Reading lessons are conducted from 9:00-^:45. a;m. in all district 
elementary schools using the Sequential Stages Reading Series, At ten 
o'clock, the first of three recesses is held, followed by the Individualized 
Computational Mathematics Learning Program, Science, social studies and art 
are squeezed in as time permits and at the discretion of the teacher. The , 
Curriculum Committee is concerned about this emphasis on reading and mathe-' 
matics; the Board has established a .policy that the most important baste 
subjects, reading and math, be taught in the morning when the students *re 
fresh and alert. ^ • f 

Around the corner, the high school campus is quiet except for the few 
students attending a 7:15 a.m. Advanced Placement Class. Two -custodians 
begin their work picking up litter after last Friday's football game with a 
crbss-town rival. Six security guards, armed with walkie-talkies, have been 
.tfcftled to the Principal's office. Due to power failure, no bells or clocks-- 
on campus are working. The guards will be used today ^to move students 
through the .six-period day. 

The high school's teachers' lounge is noisily filled with teachers 
criticizing not only the newspaper article but also the daily school bulle- 
tin, the teacher union president's position on hiring, and the' rescheduling 
of today's sports rally. The reading and mathematics teachers areparticu- 
larly concerned about the decline in the students' scores in their respec- 
tive subjects. Just a year acjo, the School Improvement Advisory Council set 
raising these scores as a top school priority. _ 

There is much unrest on this campus. The high school, with its third 
principal in six years, has recently instituted its minimum competency exams 

\ - . 17 
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with dismal results, and, there are rumors that the school may be closed 
next year due to enrollment decreases. The School Improvement Advisory 
Council has decided this year to tighten graduation requirements, teachers 
and some parents are more' concerned about what they consider the school's 
real problems, poor discipline and drugs./ This year's projected staff 
development activities are intended to .deal with several other issues: sex 
equity beyond sports and physical education, textbook and library censoring^ 
mobilizing community resources, humanizing the schools, and_ teacher evalua- 
tions under the district contract. 

The bull horn that has replaced the normal bells sounds the start of 
the first period and teachers slowly disperse to meet their Period I stu- 
dents. . Ms. Swift', the newly appointed principal, experiences some frustra-„ 
tion. Should she stay behind at school and deal with the bells or drive 
across town to the district office to attend the Superintendent's meeting? 

Does she have an option? 

Parents and community persons are watching. This is her first princi- 
palship and Ms. Swift is the object of considerable district-wide atten- 
tion. She is the first female high school principal appointed to the male- 
dominated district administrative roster. Her promotion from an assistant 
principal ship in a nearby district has not been greeted with overwhelming 
enthusiasm by those who were comfortable with things "the way they were." 
She is aggressive, knowledgeable about new instructional developments and 
willing to try out some new ideas. Some members of the community and Board 
are delighted to see someone who wants to bring a different approach. 
Others feel that the district really needs funds to do the things that are 
tried and true. What is needed is more of the same. 
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Enough about Ms, Swift, She really had no option. She sped oftf in her 
<&ar and was the fourth of 20 principals expected to meet at 9:00 a\m..with 
the Superintendent. * 

The school administrators sit in the staff lounge. Some are taking 
advantage of the coffee and donuts provided, some remark about yesterday's 
, test results news. 

It's 9:15 a.m. as the 20 prijjpipals remain in the lpunge waiting 
unknowingly for a meeting whose entire agenda has been changed. The Super- 
intendent is meeting with the two Board' trustees. These x two represent dis- 
tinctly different community interests,, but have a common concern about the- 
district test results. The Board- members insist that the Superintendent 
expand the Board agenda to include a statement from the Superintendent on 
the test scores, and a plan of how the district intends to raise these 
scores. Representing the voices of their constituents, they feel tha^the 
public is tired of excuses, and the Board members fear a *swel ling exodus of 
children away to the private schools. Likely the local television station 
will send a reporter to the meeting. ^ 

Mr. Pace, principal of the Bruhec-Elementary School for the last twelve 
years, arrives. He was stopped ,in his school corridors by 'Tim Huntley, a 
student teacher who wanted to knowwhy Bruner had sconed so low on the 
test. Mr. Pace tried to give a hurried but philosophical answer about the 
scores, making this comment: "However, young man, you must remember that 
our scores, at this school , weren't that low. In fact, we didn't change our 
scores. The papers are looking at our district performance on. the average, 

not individually aft our school. Look, I'd like totalk more, but. I 'm real ly 

/ >. 

late to an important meeting. Why don't you ask your master ^teacher?" 



\jjuntley enters ^his 3rd-4th grade .combination classroom rather . • ' 
perplexed. The students are working informally, in semi -structured groups. 
*They are tfeginning a humanities project he assigned which requires the 
exchange, of 'ideas.. in order to develop critical thinking. The culminating 
activity is a group collage which captures the essence arid tone of- the 
topic, "Communicating Differences." 

*Not seeing his supervising teacher around, Huntly states that he has 
just seen the principal and that the principal permitted, in -fact 'mandated' 
that they hold open discussion about the school , the programs and the ^sts 
that are given. Because the tests seem so meaningless to; them, the students 
stop what they are doing and begin to chat and reminisce about who cheated, 
whose pencil ..broke, who marked the wrong grids, and who faked illness. 

The supervising teacher, a ten-year veteran, hurries in with several 
copies of last night's neighborhood newspaper under "one arm and dittoed 
reading materials under the' other. She things she has a great idea for a 
new related assignment, "The Power of the Press in Communication," but 
instead, she- verbalizes her pleasure with Huntley's initiative as well as' 
the students' self-control in her absence from the classroom.. The^discus- 
sion over testing lasts about twenty minutes. "Responding to the principal's 
suggestion, the student teacher asks his supervisi ng- teacher what she thinks 
about the test scores. The teacher looks puzzled, and is at a loss about 
how to reply. How much will her answer show that she really doesn't know 
much about the structure of the test and the meaning the different scores 
haVe*for teachers? Hesitant and somewhat embarrassed, she begins a'long 
discourse on the history of the testing program. "This district and 
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community have. engaged in an extensive testing program for thfe past fifteen 
years . 

Back* at . the district office the principals arg admitted to the Superin- 
tendent's off ice at 9:30 and' the meeting begins promptly. Superintendent 
Dewey deals with a somewhat routine administrative matter and reports on the 
just-concluded meeting, with the Board .members* "If I am to make a presenta- 
tion to the Board on Monday I want- to base my comments on the perception of 
people, like yourself, who have first-hand knowledge of what is going on. 
We've been pushing basics now for a long time — yet the test scores remain 
at unacceptable levels — what Tn the world is happening?" 

A long, intense, but somewhat rambling discussion consumes the next two* 
hours. The principals' perceptions of the problem cover many factors. 

Some blame the influx of children with English as a second language and 
for whom the test is not a fair measure of their real skills — or the 
schools' instructional program. Others think that the tests are meaningless 
• to the kids and they don't try. Others insist that many students just don't 
know how to take such tests. Some respond with questions, such as: Does 
this test measure what we teach? Do we know which specific students or 
groups of students account for the lowest scores? Are all the teachers 
really implementing the new instructional program? Some suggest solutions 
such as more inservice training, or changing the tests, or not releasing the 
test scores. 

The meeting ends with a marked feeling of general frustration. Ms. 
Swift, thinking about the forthcoming meeting, remarks that there are times 
when it is nice not to be a superintendent. 
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The Administrative Assistant to the Superintendent picks up the extra 
xeroxed copies, and gets coffee for her boss. It has been a busy day 

. • J . ^ 

already. 

» • .. 

It's 1:30. Chris is concluding a telephone call a^s ttie eight members 
of "hi ^executive cabinet enter the Superintendent's office. Sitting around 
the large executive table, from left to right, fere: the Director of 
Curriculum, the Administrative Assistant to the Superintendent, the District 
Personnel Di rector,- the Evaluation and Testing Coordinator, the Director 
of Pupil Personnel Services, the Director of Federal Projects and Programs, 

the District Budget DirectoV, and the Associate Superintendent for Elemen- 

* ■" 

\ tary and Secondary programs. 

Staff assistants,. have brought additional school and district documents 
* including last year's unused Title I Evaluation .and Reporting System 

(TIERS), the basal readers used at each grade level, the mathematics learn- 
ing packages, and ihe computer outputs presenting school and grade level 
information, as well as an item analysisfor each question administered 'to 

« 

the students. » 

ft 

The Superintendent has described the unexpected meeting with the Boa>d 
members and the meeting with the principals. The chief administrator out-, 
lines what the Board is expecting and he asks them one by one to give his/ 
her perceptions on the problem and any ideas he/she might have both as to 
solutions and as to how to handle the Board presentation. But not everyone 
sees the prdolem the same way. 

The Curriculum Director starts off. "I hope we don't panic over these 
latest test scopes. After all, our new curriculum emphasizing basic skills 
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has only been in operation for a little more than one year. You can't ex- 
pect that things will change overnight. These things take time, f^y advice 
is to keep emphasizing what we afe do,ing'-- we have -a 1 sound plan for getting 
kids to study, more basic skills, fty advice is to* say that this is a tempo- 
rary condition^ — things will improve with time." 

"Besides," the Curriculum Director continues, "I'm being plagued by 
self-appointed book censorship groups who are insisting that we. clean out 
our texts and school library of what they consider offensive materials. 
When this issue reaches the Board — then we will have a real .problem!" 

The District Budget Director, an accountant by training, sees the 1 
problem as one of ineffective teachers who continue to do an inadequate job 
in the classroom. "Wty not," he argues, "begin aggregating these test 
scores by teacher and see if any teacher continually has classes of students 
.who fall below a pre-selected percentile? These teachers, once identified, 
could be^given extra help and if they still perform below district stand- 
ards, then we can 'counsel tifrern out of teaching.' In the business world we 
do this all the time I mean, we look at; a salesman's monthly records. If 
he can't perform — out he goes! All the new curricula and regulations and 
media don't add up to anything if you have weak teachers. I say, use the 
data to attack what is obviously the most serious problem -- some poor 
teachers!" Most in the room shift uneasily in their chairs. 

The Coordinator of Testing and Evaluation attempts, to cool what might 
become a heated discussion. "At the risk of repeating myself endlessly, let 
me say that I warned you some time ago that this might happen. As you will 
recall, I predicted that there was a mis-match between our present norm- 
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referenced tests and our newly developed curriculum. As it is now, the 
tests simply do not measure what we are teaching. At the time we changed 
our curriculum we should have looked at a different, more relevant norm- 
referenced test. But, I'm afraid it is too late to do much about it right 
now. Any shift we make will immediately raise suspicions that we are 
selecting an 'easier' test, just to make us look good." 

"Having said that, let. me suggest a more immediate solution ,. namely 
teaching the kids to take tests better. I don't mean anything illegal, like 
givin^-them the test answers. Don't get me wrong. But many teachers report 
that^ome/k)ids are really frustrated by the Whole process. Indeed, some 
kids are forced to take the test even when they can't read English. I ask 
you - of what use are those results? What are we doing! Also, some kids 
are confused by the standard answer forms. I say let's have some teacher 
inservice training on how to teach kids how to take tests, and let's remove 
those test scores of kids who obviously cannot read English very well -- or 
better yet, let's get the test translated into their language. These kinds 
of actions will' bring more immediate improvement - something we desperately 
need." * 

The Superintendent's Administrative Assistant, who normally doesn't say 
too much at these cabinet meetings, unexpectedly speaks up. "It seems to me 
that we are not the only district with this problem. Why don't we find some 
other districts that are like ours in size and student make-up, etc., and. 
see what they have dope? If they have come up with something that works for 
them ~ let's adopt the same plan here. Such, an approach will save money, 
time and will be successful. -Those are words that the Board likes to hear." 
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Finally, the Associate Superintendent for Elementary and Secondary • 
Schools speaks up, "At the risk of further complicating what has been said 
and suggested, let me remind you — us -- that the problem is not equally 
distributed in the district. Our elementary school scores did not decline 
but.the secondary ones did. Also, some elementary schools actually 
increased their test scores. As we struggle with this problem we should be 
mindful of these facts. No' simple, single strategy is going to solve this 
problem, even if the board and some segments of 'the public are clamoring for 
one. We must *fcery carefully think through a long-term, comprehensive 
strategy that will get at the core of the problem — otherwise, we will- be 
holding this same discussion, again this year that is, if we are still 
here." 

The Budget Director reminds the* group that any plan must be developed 
within the confines of yet another budget cut. There will be even less 
money for staff development, evaluation studies, and the like. 

At this point, the Superintendent turns the discussion to the district 
enrollment and budget projections and the administrators submit plans they 
have developed to accommodate the anticipated cuts. 

At 4:30, the Superintendent thanlcs them for their perceptions, ideas 
and advice, and ushers them out of the office. The secretary buzzes and 
reminds the Superintendent that he will have to leave in an. hour in order to 
make a presentation at the district's student citizenship dinner. 

If's now 9:15 p.m. At home," the superintendent sits down, legal pad in 
hand, to sort .ojut what has been said. The several people to corner him at 
the citizenship dinne^ about the test scores made it clear that a potential 



crisis is brewing. What he says at the rapidly approaching Board meeting 
takes on increasing importance. He looks at his notes, searches his memory 
and begins to categorize the suggestions made. The following appear. 

1. The test approach \ , s 

0 select a different test 

° teach the kids to take tests 

° relate the test content to the curriculum 

2. The teacher approach 

0 improve teacher skills 

0 identtfy incompetent teachers — and provide inseryice 
traini ng 

0 fire those who can't meet district standards 

3. The planning approach 

0 develop a comprehensive plan that considers the several 
dimensions of the plan 

4. The adoption approach 

0 identifying districts that are similar to ours 

0 sort through their respective plans and pick one that 

has been effective elsewhere 
0 implement it in our district 

Somewhere among all these approaches, there lies an answer to the 
district's dilemma — but what is it, and how does he begin to find it? 

Discussion 

While the details might differ, we suspect that there are 
characteristics or parts of this scenario that are familiar to many 
administrators in school districts across the United States — 
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° districts that collect test, data and find those data released to the 
public through the media;,, 

° districts in which there is a public outcry when the test data do 
not reveal adequate pupil learning and achievement; 

° districts that are urged, or motivated to "do something" to bring up 
the test scores; - ^« 

°- districts where there are multiple and differing perceptions about 
what causes the problem and, accordingly, about what will best 
achieve the desired goal of raising test scores. 

Such districts face a very real challenge and dilemma because they are 
bften confronted with data that challenges their credibility to perform * 
their central instructional role, because it is widely assumed that test ^ 
scores accurately reflect the school's instructional program. The public 
often assumes that there is somie causal relationship between the district's 
instructional program and test score fluctuations. Often the public de- 
mands, and the district officials promise., that the district will make 
necessary instructional program adjustments so as to reflect increasing 
rather than declining pupil^schievement. 

But what should be done? This, of course, rests on knowledge about the 
district's instructional program and its relationship to the tests used to 
measure instructional effectiveness. In many districts this relationship is 
simply unknown or orj/Vy vaguely understood. The districts have not examined 
the relevance of th£\testing program to the instructional program. .Indeed, 
many districts would be hard pressed to describe the degree to which their 
instructional program is being taught by district teachers behind their 
classroom doors.* Thus, the districts, such as the one portrayed in our 
scenario* can be characterized as possessing instructional ly relevant data 
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but data that, in its present form, is not being adequately utilized as a 
potential management tool. 

This book is intended to help school administrators, board members and 

teachers to better understand the potential that district testing and eval- 

4 

uation programs have for improving the instructional program. Actually, our 
purpose goes considerably beyond just focusing on testing and evaluation as 
isolated school district functions. We conceive of testing and evaluation 
as being components of a district instructional management information 
system . 

To" elaborate, we define a district instructional management information 
system as follows: 

District -We consider instructional improvement as a school district re- 
sponsibility. To be sure, the individual teacher in the classroom is 
the basic component of the instructional program, but this does not 
relieve the district from full responsibility of seeing to it that the 
teachers are competent, using appropriate instructional methods, and 
furnished with necessary supplies, equipment, and instructional mate- 
rials. Teachers alone cannot carry the entire instructional burden; 
neither should they assume the total responsibility and blame for the 
instructional program's shortcomings. This responsibility rests with 
the school district., A 

Instructional - By this we mean the delivery of the district's curricula. ~^ 
It includes the teachers' methods of determining learner needs, in- 
* structional decisions and activities, and assessments of the effective- 
ness of instruction, including district -wide testing and instructional 
program evaluations. 

Management - This includes the planning, developing, implementing and 

assessing the district's instructional programs. It involves coordi- 
nating many components ranging from personnel. to budget to school site 
administration to inservice training. 

» 

Information - This includes data collected about such topics as pupil 

achievement, teacher behavior and activities, program implementation. 
It includes using ^any kinds of tests, program evaluations, opinions 
and attituca surveys. 
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System - This recognizes that the various components, e.g., instruction, 
test scores, evaluations, inservice training, have a relationship to 
one another and that they t shou!d not be isolated one from the other. 
For example, a change in the instructional program in a given subject 
will, or should, have some impact on other components, e.g., tests, 
inservice training. 

Put another way, we propose that districts can more efficiently and 
effectively manage their instructional programs if they have developed a 
system for relating information and data they have gathered to their * 
i nstructional program. 

Our scenario district is rich in data, but poor in not having developed 
a management system that will allow them to use those data for instructional 
program improvement. This book is about Jiow such a district instructional 
management information system can be developed. 

We think such a book is needed at this time. Schools and school dis- 
tricts are not doing very well. That is, public schools are increasingly 
viewed by the public as not being very effective instutitions. While much 
of the present crisis in public education results from nationwide, indeed, 
worldwide, economic decline and political priorities that have shifted away 
from public education, one can argue that public schools must present evi- 
dence that they are effective organizations before the public will express 

much confidence in the schools. A popular public perception is, that merely 

C 

"throwing money" at the schools will not necessarily improve the quality of 

<> 

the schools. 

Experience has shown that single element strategies just don't work. 
For example, some have advocated massive inservice teacher training. This 
strategy assumes that the individual teacher behind. the classroom door is 
the problem. There is some merit to this approach, but in our view, it is 

ERIC 23 



- 25 - 

{ 

t 
\ 

too narrow. It does not address the systematic and district-wide responsi- 
bility for instructional improvement. 

Others have advocated a data-based accountability approach in^which ^ 
test data, are used as a prod to get teachers to be more responsible — and 
therefore, ultimately more effective. This hasn't worked either; it has 
resulted in the mis-use of test data as a punitive device which has resulted- 
in deception and evasion and a lessening of trust between teachers and their 
supervisors? Again, this strategy is not based on any comprehensive, syste- 
matic instructional improvement system. 

We believe that what is needed is a system or approach that will link 
together our advancing knowledge about testing, evaluation and^instruction 
within the complex setting of school districts. If done properly, it can 
provide one approach to improving school districts' instructional programs. 
Clearly, it is not a panacea. It has its limits. For example, there are 
important school activities and outcomes that are very difficult, if no^"^ 
impossible to capture or measure in a testing program. We are talking afoOTf"" 
such outcomes as the students' self-confidence, or appreciation of beauty, 
or enthusiasm for inquiry. Thus, one has to be careful not to begin dimin- 
ishing the provision of these important activities on the basis that they 
cannot be adequately measured, at least not with our present level of test- 
ing technology. Frankly, we doubt that some of the schools' most important 
outcomes can ever be measured, except in a very subjective way. But this r 
doesn't lead us to conclude that all measurement and evaluation of perfor- 
mance is undesirable. We must do what we can, be aware of the limitations, 
and take actions necessary to see to it that' the schools more clearly rea- 
lize their instructional potential. 
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CHAPTER 3 



School District Questions 

in which four simple questions implying 
."quick fix" solutions are reframed for more 
productive thinking . . . 
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CHAPTER 3 
Introduction 

? 

t 

/ 

I 

The reader will recall that we left the superintendent of our scenario 
scnool district at home, late in the evening, legal -sized pad in hand on 

» K 

which he had written the four approaches he had heard about the looming cri- 
sis during his conversations with his principals and his cabinet. The four 
approaches were: the, test approach, teacher approach, planning approach and 
adoption approach. Our superintendent, and others who may find themselves 
in a similar situation are indeed faced with a formidable challenge. The 
crisis is real.. That is, the public believes that, at least as measured- by 
the district's testing program, the district simply isn't educating children 
to acceptable levels of learning. What is more, the decline in test scores 
seems not to have been stopped by the sf forts already undertaken by the 
superintendent* and his administrative and teaching staffs who care about the 
same problem. Not only do they care, they have acted. They have formed a 
district community /parent committee to define district goals — perhaps in 
the hope that if the* goals are more clearly explicated, they will also be 
more obtainable. Yet, in spite of concerns and effort, the district remains 
plagued by unacceptably low test scores which presumably represent weak 
student achievement. ^ 

But the superintendent's dilemma is not Wily that the district's 
efforts in the past have not worked. He is also challenged by the four 
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suggested approaches he has recorded on his note pad. Each has the "ring of 
truth: about it. Does the answer to his district's dilemma rest on follow- 
ing one of these approaches? Or some combination of approaches? -Or in 
going in an as yet unidentified direction? The board expects an answer. He 
and his staff want an answer. Where does he begin? 

We suggest that he begin by understanding the myths and the facts 
underlying each of the questions. 

j. Richard Harsh,* the author of this chapter, deaws upon considerable 
training and experience in tackling the difficult task of analysing these - 
approaches fairly and concisely. For many years, he-was the California 
field director for Educational Testing Service. This brought him into a 
working relationship with many school district instructional, testing, and 
evaluation programs. Since retirement, he has served as a much-sought-after^ 
consultant who specializes in district testing, evaluation and instructional • 
problems and programs. In writing this chapter, he has drawn extensively 
upon his many years of experience and the insights they have provided. 
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r. How Soon Can We Expect Test Scores To Go Up? 



The publication of district test results in the local newspaper often 
sets off a strong reaction in the^communi ty. The board of education may 
receive strong demanHs from vocal citizens to initiate reforms in the in- 
structional program that would improve the achievement of the students. 
Special funds may be allocated to schools to improve the test results. Such 
a flurry of activities stimulates people to question past and present prac- 
tice anc ! ask what>esults may be expected. 

"How soon can we expect the test scores to go up?" 

This question can't be answered without knowing which tests are refer- 
red to. However, as this is mentioned, the quick retort is often — "the 
tests in reading, mathematics, and language that have always tJeen given." 

Myth : All tests with the title of "reading test," measure reading in 

the same ^ay, or, all the tests of mathenfatics measure the same 
skvlls and Knowledge of mathematics J 

Fact: There are a variety of tests that are given to measure stu- 

dents' skills in reading, math and language, and the various 
. tests (by design) have different relationships to the local in- 
structional program. Various tests of reading, math or lan- 
guage may ask students to demonstrate different skills, know- 
1 edges and applications * 

Discussion: There are many categories of tests. Each category of tests 

reveals different things about student learning. And, within 

each category of tests, the tests may differ from one another.. 

The following array of reading tests may be used in a school 

district. J 
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Published Standardized Reading Tests 

These tests are developed, normed and published by commercial test pub- 
lishers for sale to educational institutions\hroughout the nation. They 
provide norms constructed by the publisher that are derived from testing a 
sample of students in various -types of school systems that the publisher 
believes to be a representative population of the students in .the nation. 
The questions on these tests are designed to measure the skills, knowledges 
and applications that are believed to be most ' generally common to many read- 
ing programs throughout the nation.- These tests may be considered as mea- 
sures of generally common reading skills with norms that suggest the ranking 
of levels of performance on the test. 

The California Achievement Tests, the Stanford Achievement Tests, the 
Iowa Tests of Basic Skills, the Sequential Tests of Educational Progress, 
the Metropolitan, Achievement Tests, are examples of these standardized tests 
tJhat have sub-tests to measure the skills of reading, mathematics and lan- 
guage that are considered to be generally common to many school programs. 
Analyses of the reading tests of these publishers reveals that the content 
and skills measured by the questions differ from one test to another. For 
example, reading vocabularyMs measured by some tests (CAT, SAT) by present- 
ing a word and asking the student to jdentify from among four options that 
one which means the same as the word given*. In contrast, the Iowa Test of 
* Basic Skills measures reading vocabulary^ asking the student to identify 
the' meaning of a word in a particular sentence or ^stefy. E.g., 

Bob said, "I'm glad that I can bank on your loyalty*" In this 
senteffce the word BANK means : a J a place to keep money; b)*the 
side of a stream or river,; c) to.be able to depend upon 
another; d) to arrange a stove or fire. 

0 . . 
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State Assessment Programs 

These t,ests are designed under the direction of state educational agen- 
cies to provide measures of the skills, knowledges, and application that the 
state requires in the curriculum^ or that are expected to result from the 
instruction in all districts of the state. Some states may construct their 
own tests (California, New'York, etc.) while other states may contract with 
commercial test publishers to dev.elop tests to their specifications (e.g., 
Florida, Michigan, etc.). $ 

Norms for these tests are developed from the performance of the stud- 
ents in the particular state. The norms may present numerical ranking of 

schools within the state, or percentile ranks for different levels of per- 

\ - i 

formance by individual students, or may provide numbers and percents of 
students within each school or district that demonstrate competence in each 
skill according to the standards established by .the state agency. Some 
states (California) provide norms for groups of schools with similar demo- 
graphic characteristics such as mobility of students, socioeconomic condi- 
tions, percent of minority population, etc. The different norms both within 
and between state assessment programs provide different information regard- 
ing the performance of individual students, sub-groups of students or the 
schools and districts being assessed. 

The use of the state testing results also differs in relation to the 
design of the tests and the manner in which students are tested. In some 
states, all students in all schools take the same test on the same day of 
the year. In contrast, the California State Assessment program is not de- 
signed to provide information about individual students. California uses a 
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"matrix sampling" design which provides many different short forms of the 
assessment of basic skills in reading. The combination of all forms of the 
test covers all the skills that are desired to be measured, but any single 
form will measure only a portion of the skills. Students take different 
forms of the test that are randomly assigned within each school. The scores 
from all forms of the test are reported for schools and districts butyio 
scores are available for individual students. The report made to districts 
3 in Ofirt-rfornia provides a "percent of correct answers" to the composite of 
all skills {neasured by. all forms of the tests. Sqch a report provides 
information of the percent of students in a school or district that are 
successful with elach skill measured. This information is quite different 
from the state reports that provide percentile norms for each student or 
school population, and neither of the state reports may be compared to the 
national publishers' norms. 
Proficiency Tests For High School Graduation 

By 1980, thirty -eight states had adopted legal requirements of profi- 
ciency testing prior to granting a^high school diploma.Vrhe designation of 
competencies and the standards of performance were made by state departments 
of education in some states while other states mandated local definition of 
the competencies to be measured and the level of performance required for 
high school graduation. The most common practice was to consider these 
assessments as measures of th* minimum essential skills 1n' s the language arts 
and mathematics that would be required by out-of -school daily living. In 
California, where the law required local definition of competencies, such 
designations were made by district committees of citizens and educators. 
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The tests commonly measure skills in reading, math arid writing that are con- 
ceived as essential for coping with post-secondary life. Thus, the content 
measured by these tests typically covers only a small portion of the read- 
ing, math and language curricula of a district. 

A wide variety of instruments have been used to assess the minimum 
essential skills for high school graduation. When a common test is required 
by the state department of education, some states constructed new instru- 
ments, contracted with testing companies for unique development, or adopted 
a commercially prepared test of basic literacy. In those states such as 
California where the responsibility for the assessment was delegated to the 
local district, a wide variety of tests have been constructed locally, or 
groups of districts have agreed on common definitions and constructed, con- 
tracted for, or purchased commercially prepared tests. 

The proficiency tests for high school graduation vary widely between 
and within some states both in terms of the skills measured, as well as the 
level of performance that is required. In the context of local definition, 
the test results are meaningful only in terms of the particular ski 1.1s ?nd 
levels of performance that have been adopted. The results of these tests 
thus may have very different relations to the local curriculum, the state _ 
curriculum, or nationally standardized tests of comprehensive measures of 
the common instructional program. 

Due to the fact that the legislation requiring high school competency 
examinations frequently assigns penalty to students not performing at a 
specified level (denial of diploma), high attention has frequently been 
given to devising instructional packages to insure that the particular 



skills measured by the test were mastered by all students^ Increases in 
student test scores in this context may be achieved more rapidly than 
increases in test scores that assess the full spectrum of the school 
curriculum. 

Elementary Proficiency Tests 

These tests have emerged in many states and regions as extensions of 
the legal mandates for demonstrated competence in the basic literacy 
skills. As secondary districts were confronted with the wide range of aca- 
demic achievement of entering students there was concern expressed for the 
academic development attained in the elementary schools. 

A range of strategies for the definition of essential competencies 
includes local district, counties, states, and regions specifying contents 
and levels of performance that should be attained by the middle or uRper 
elementary grades. In some regions of states such as California that re- 
quire local definition, there are attempts to articulate required minimums 

from the elementary through the secondary schools. The concept behind the 

\ 

design of these tests is that certain essential skills should be attained by 
.all regularly progressing students by a particular age or grade. 

Generally, these tests, as is true of the high school graduation tests, 
are designed to measure the essential skills in reading, math, and language 
that are locally defined as necessary for continuing educational progress. 
The questions, are of a limited range of difficulty that assesses a partic- 
ular level ol| competence expected at the end of a particular educational pe- 
riod, such as sixth grade. The tests do not measure the full scope of the 
curriculum in mathematics or language arts, but provide monitoring informa- 
tion concerning the number of students that have achieved the essential 
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skills by the upper elementary grades. The fact that districts may define 
the skills and criteria of mastery makes it difficult to compare the per- 
formance on these tests with nationally normed tests, or state testing 
results. 

Curriculum-Referenced Tests 

These tests are intended to measure student mastery of the content and 
skills of a particular curriculum. Some of these tests are constructed by 
the publishers of curriculum materials to measure the mastery of that par- 
ticular content. Other tests may be constructed to measure the curriculum 
developed by a local educational agency. The locally constructed measures, 
are designed to assess the student attainment of the objectives of the local 
instructional program. A wide variety of criteria are used to interpret the 
scores made on the .curriculum referenced tests. Some use the percent of 
students answering each question or group of questions measuring a particu- 
lar skill while others develop local percentiles to array or group the per- 
formance of the student populations. 

These tests are probably most directly related^to what students have 
experienced in the instructional program. Insofar as the tests systematic- 
ally assess the content and objectives of the local program the results may 
be considered to be relevant to the question of the degree of mastery that 
students have attained. On the other hand, the local program may have 
unique contents, instructional materials or unique sequence and emphasis on 
certain skills that may diverge from what may be emphasized in another dis- 
trict or what may be measured by external .measures such as state or nation- 
ally standardized tests; ^^^^ 



ERLC * (\ 



- 36 - 



Tests. Differ F rom One A n other in What They Tell About Student Achievement 

° Nationally standardized tests are assessments of broad general out- 
comes that are conceived to be quite common to many educational pro- 
grams. The norms provided by the test publisher provide a method of. 
ranking local student population's performance in relation to a 
sample of students thought to be representative of the national 
student population, S 

° State testing programs are designed to measure those skills and 
knowledges that are judged to be the common desired outcomes of* 
instructional programs in the state. The norms or summary statis- 
tics provide a means of comparing local student performance with the 
total student population of the state. 

°* Proficiency tests, especially those required for high school gradua- 
tion, tend to' emphasize minimum essential skills that are thought to 
be necessary' for the demand of. post-secondary school; life. Student 
performance is interpreted by applying local standards of the dis- 
trict or the standards established by state mandates. Elementary 
proficiency tests tend to reflect local designated skills and know- 
ledge that are deemed essential to further educational progress. 

° Curriculum-referenced tests measure the attainment of the content of 
a specific curriculum or local instructional program. Interpreta- 
tion of the student's performance is made in relation to the extent 
to which the student has mastered those objectives of the local 
instructional program. 

"These explanations are informational but the question originally asked 
still persists — What can we do to make the test scores go up? The low 
scores that have been reported for the district tell us that the students , 
aren't learning what- they should!" 
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Myth : If the group scores of a district go down, it automatically 

. means that students are learning less than ip previous 
periods . 

Fact: Many different factors can account for changes — either up or 

down —in test scores of the student population of a 
\ district , • ^ 

Discussion: District test results are usually reported as the average 

score of all students such as the M, 38th percentile," or "forty 

percent of the questions were answered correctly, 11 or "the di s- 

trict ranks below sixty percent of the districts in the state." 

Such summary characterization of the performance of the total 

student population often conceals more than it reveals . These 

quantitative summaries of the central tendency or average of 

the group do not tell the number or proportions of students 

making scores well above the average reported, nor do they tell 

about extremely low scores that may have great effect on the 

average of w the total population. 

Persons with anxious or critical disposition are prone to 
take the "average; score" as the characterization of all stud- 
ents. It is possible that the average score reported was made 
by no student. (I have two sons. One is 5 feet tall, the 
other is 7 feet tall. . The average height of my two sons is 6 
feet — this doesn't provide an accurate description -- only a 
numerical characterization of the average height.) 

However, regardless of the method of reporting test per- 
formance, test scores do go up and down. With the changes in 
the reported scores it is appropriate to ask — "What are the 
factors that might change test scores. 11 
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The mobility of the population within regions and the na- 
tion produces rather dr.amatic changes in the chara_c£eri sties of 
the students in many districts. As occupational opportunities 
shift in location there are related shifts in the population 
which works in the occupations. The development of a large 
technical industry in a particular community may bring a new 
population of professionals and technicians. A residential 
community that is changing to diverse industrial plants may 
have a shift in population as the workers required may have 
different education and socioeconomic status from the former 

population. . 

The changes in the population of a school attendance zone 
may produce shifts in the abilities and achievement of the 
student population. For example, in a period of two years, a 
school district increased its enrollment by 25 percent and also 
observed an increase in the median prcentile score in reading 
from 51 percent to 64 percent. During the same two years, , 
another district in the region experienced 50 percent of the 
population moving to other locations and a new population of 
equal size but different demographic characteristics moving 
into the district. In this district with high mobility the 
test scores showed a substantial decline. 



Test scores may rise or decline because of a change in the 



test scores may be related to either variations in the norms of 



tests that are used to measure the basic skffls. 




e change in 
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the tests, variation in the types of questions used by dif- 
ferent tests, variation in the content and skills required by 
the test questions, or a combination of all of these factors* 
Such changes in test scores are often related to the degree to 
which th'e skills measured by the test are congruent with the 
content, sequeACe "and emphasis of the local curriculum. 

Pressure to raise test scores has sometimes resulted in 
inappropriate activities. As mentioned previously, the summary 
report of test scores typically presents the average of the 
scores of the total student population. Test score averages 
may be raised by the unprofessional action of "selecting a 
sample 11 that omits those students who are known to be low / 
achievers. A similar distortion of the average test results 
may be produced by the lack of effort to insure that all stud- 
ents are tested. A regretful result, of such a strategy re- 
vealed that only two thirds of the population of a high* school 
were present for the te$t administration. 

The concern for raising test scores has produced a wide 
variety of procedures to prepare students for the test they 
will take. The most inappropriate procedure is teaching the 
exact content and format of the questions in* a standardized 
test. When persons have t^een accused of "teaching to the 
• test," some have defended their actions by saying that they 
were merely using practice* exerci ses to develop student "test 



wiseness" or familiarity with the format and the type of ques- t 
tions contained in the standardized test.' While there may be 
appropriate* activities to familiarize students with test-taking 
skills, there is no question of the inappropriateness of . re- 
vealing and teaching the actual content, of test questions. 
Such actions reflect a total lack of understanding that the 
questions on a test are only a sample of a very large domain of 
skills, content and application. The teaching of ^the partic- 
ular content of test items that are to represent an entire 
domain leads to the consequence that that the test scores are 
erroneously raised. The students^^-ai^further injured by the 
assumption that their test .scores suggest that they have 
achieved particular skills and knowledge of a domain when they 
have not. 

The consequences of strategies designed to rapidly raise 
test scores by inappropriate and unprofessional activities are 
very serious for both educators and students. Moreover, the 
accumulated data base on test performance that currently 
exists suggests that the inappropriate activities will be 
discovered within a period or Weeks or certainly by the next 
year. ^ 

The combination of sincere and appropriate efforts to im- 
prove student achievement has high likelihood of bringing about 
a rise in some test scores. At the same time, it is useful to 
consider whether the rise or decline of the average test score 
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has any predictable significance for individuals in a partic-. 
ular class in one school, for boys or girls, for students with 
limited English proficiency, or for particular ski 1 Is such as 
vocabulary, solving problems, or applying the skill in out-of- 
school situations. 

In a school or district with several hundreds or thousands 
of students, an increase in the average score provides little 
assurance that a* particular students score will be higher or 
lower. For example, concentrated effort with the most severely 
retarded students may result in an increase for them, while the 
most able or accelerated students may show no gain or even a 
decline. When an average is computed for-a large population 
the increase of tests scores for those who had previously made 
veVy low scores will raise the average score but there may be 
no improvement for the more able students. The converse may 
also be found: emphasis given to the, group of high achievers 
may raise th^ir scores while the less able show no improvement 
or a decline. 

Sub-groups, within a school or district student population 
may not demonstrate the_ same increase or decline that is sug- 
gested by shifts in the average score. It is not uncommon' for 
individual schools in a district to have average scores that 
are above or below the average score for the district, and eth- 
nic sub-populations often show substantial divergence from the 
district average. 
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The increases in average scores in reading, math or lan- 
guage do not provide assurance that all skills within these 
areas of the curriculum have improved. It is quite common to 
observe an increase in the average math scores following con- 
centrated intervention or emphasis on path computation. . How- 
ever^ it is often observed that while the computational skills 
have improved, thet'e may be no comparable improvement in the 
ability to solve word problems or apply the skills in out*-of- 
school situations/ 

These examples of the many factors related W^the rise or 
decline of average test scores suggest that careful analyses of 
the congruence of the content measured by the test and the 
local curriculum as well as systematic analyses of the needs 
and characteristics of sub-groups of the total student popula- 
tion are essential for understanding the meaning of test per- 
formance. 

Such, explanations- are helpful to incease understanding, 
but the interested parties involved in education will persist 
with the conviction that improvement in the educational process 
will improve children's learning and these changes will be 
mirrored in the improved test scores. 

. M e can dramatical l y affect children's learning and have the 
' change ref lectio 1 jnri sing test scores it the schools do things 
right . " " • 

This is a true m yth. However, it is a very complex process to 
enhance studen t l earning in the comprehensive goals of the. 
curriculum"alTd~ d^mon strate that change through test scores . 
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;ion: Longitudinal sCtrcfre^ of the acquisition of skills and knowledge 
of the population served by the schools suggest that very great 
improvements have been achieved since the beginning of the 
twentieth century. Statistics on literacy, the knowledge ex- 
plosion in the sciences, the field of automation and. the num- 
bers of students graduating from the various levels of educa- 
tion attest to these improvements. 

There is need to recognize that education is a process 
that is constantly being affected by social and cultural forces 
that restrict or facilitate the learning of individuals. The 
learning of an individual includes the internalization of^in- 
terests, work habits, aversipns/ attractions, values, social 
participations, as well as the skills, knowledges and applica- 
tions of the academic curriculum. Individual learning is 
unique to that person and each person has pressures and demands 
that establish priorities and time schedules for the various 
skills, knowledge, and participation that will be attended to 
within any time period. The construction of an instructional 
program that will be effective for the diversity of a hetero- 
geneous student population must be planned to enhance Indivi- 
dual learning with all the diversities that naturally exist 
between persons. 

Hun^n learning cannot be designed as a factory might de- 
sign the processing of inanimate materials into products such 
as cars, can openers, or washing machines. Thus, while there 
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may be often-stated broaji goals that are to be realized through 
education, the translation of these goals into the relevant 
contents of study and the design of instruction for individuals 
presenting wide differences at entry to the school classroom 
creates a very complex task. 

The assumption that what a student knows and can do may be 
directly assessed by a test of a sample of questions of a very 
large domain is open to man/ireservations. All persons can 
recall circumstances in which they failed to answer a particu- 
lar oral or written inquiry within a field of knowledge or 
endeavor that 'they were functioning in with satisfaction and 
commendable productivity. Such occurrences are similar to the 
concerns described in a previous section with the congruence of 
the content and format of test questions with the curriculum 
and instruction the student has experienced. 

Enthusiastic support may be given for the assertion that 
children's learning may be improved through careful planning 
and implementation of an instructional program. The assessment 
of the outcomes anticipated from the program must be designed 
to insure that there is reliable and valid evidence of the 
attainments of the content and outcomes of the program the 
students have experienced.. 

Identification of the strengths and weaknesses c£>*the 
various contents of the curriculum, of various sub-groups that 
compose the total student population, as well as existing and 
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desired strategies of instruction are important actions to pre- 
cede the development of an instructional plan that will improve 
the tested performance of the students in the district. 
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1 I . Why Don't We Identify Incompetent Teachers 
and Do Something About Them? 

This question is frequently asked of administrators by board members, 
district advisory, committees or individual parents who are concerned with a 
wide range of conditions that might range from increasing school budgets, 
vandalism in the school and community, low test scores published in the 
newspaper, discipline problems in the school, reports of academic deficien- 
cies of local students applying for college admission, etc., etc. 

The statement of the question frequently does not identify the problem 
or condition that will be solved by "doing something about incompetent 
teachers." Moreover, there are varieties of sub-elements or., sub-questions 
that need clarification prior to a response or development of a plan for 
discussion or action. Certainly an initial step for clarification is the 
recognition that the question as stated asks for two actions — identifying 
incompetence and then doing something to eradicate it. Identification of 
competence (or incompetence) of teachers requires a detailed analysis of the 
personal and professional knowledge, skills, attitudes and behavior that are 
needed to implement the instructional program specified by the local board 
of education and administration. It is also apparent that this request to 
eliminate incompetent teachers stems from the belief, by the person asking 
the question, that an undesirable event or condition is directly and solely 
the result of certain persons 1 incompetence. Of equal importance in re- 
sponding to this question is the need to identify the evidence or criteria 
that the persons asking the question are using to draw the conclusion that 
there is incompetence among the teachers. 
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The statement/question — "Why don't we identify incompetent teachers 
and do something about them?" — needs additional discussion and clarifica- 
tion to identify the problem and the evidence that prompted the question* 
Since there is no generic list of characteristics that identify an incompe- 
tent teacher, answers to further inquiry are indicated. To examine the 
postulation that there are incompetent teachers the following questions may 
be in order. 

° What are the indications of an incompetent teacher? 

° Is the incompetence only noted with students of particular 
achievement, motivation, cultural background or age? 

° Does the lack of competence appear to be found at certain 
grade levels or with certain subjects of the curriculum? 

° What criteria or evidence is available and used to document 
the competent and the incompetent teachers? 

° What causes children not to learn what is expected? 

° Are there legal, political or institutional constraints on 
the actions that may be taken to deal with the incompe- 
tent? J 

The response to inquiries for clarification of the problem and the evi 
dence or condition that promoted the question of dealing with incompetent 
teachers often includes reference to the reported test results of the dis- 
trict. This was the case when a district advisory committee was formed to 
investigate the continuing decline of the test scores of the local schools 
and presented the following questions to the superintendent of schools. 
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"Can you obtain the t.est results for each teacher's class ' Jp 
or classes?" s 

"Can you compare the results for all teachers and then 
determine which teachers get high or low achievement?" 

"Wouldn't this be a good way, to identify the incompetent 
teachers?" 

These questions concerning the use of test results should prompt the 
further question: Can --or should — test results be used as a single 
criterion of effective teaching? Knowledge of the multiple causes for 
behavior and learning, coupled with the longitudinal accruals throughout 
a person's educational experiences indicates that although test results 
might be used in this manner it would be very inadvisable . However, per- 
sons proposing such a criterion for identification of competent teachers 
do not readily accept the reply that such a procedure would b4 inappro- 
priate without further explanation* Examination of the proposal indi- 
cated by the question reveals several myths that provide the basis for 
the erroneous assumptions. 
* 

Myth : This yeac's test results for each class or grade- are a direct 
indication of what the teacher~has taught and what the students 
have learned during the year . 

Fact: There are many different types of tests which may be inaccu- 
rately related to what, has been taught in a particular class or 
school . ~~ 

Fact: Most tests measure what has beenl learned in previous years' as 
well as those outcoma^that are expected for a particular age 
or grade"! — _ 

j 

Myth : Standardized norm referenced tests indicate how well the 
teacher has taught and how -well tie students have learned 
common skills and knowledges. 

Fact : Nationally standardized tests have no exact match to what each 
teacher in each school has taught. 
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The teacher having a class with the highest test scores is the 
most competent teacher , ~ 

Student populations may have very different levels of achieve- 
ment at entry to a particular teacher's class, me test re- 
suits for one particular year are not solely a reflection of 
what has been achieved in that year . 

Interest in using test results to evaluate the effectiveness or 



competence of teachers is plagued by the/"inexact match between 
what the teacher has taught and what is measured by the tests. 
It is well known that the nationally standardized tests are de- 
signed to measure the outcomes of instruction that are gener- 
ally common to most districts throughout the nation. Thus, 
while there is a general match for the most common outcomes of 
instruction .there is an irregular and poor match with the con- 
tent of instruction in some classes and schools of the nation. 
Moreover, while the general outcomes may be common goals for 
many or most schools in the nation, there may be wide differ- 
ence in the sequence and timing of the particular skills that 
are required to attain those goals. 

The use of locally designed tests that are intended to 



assurance that the results of any single testing will identify 
effective and ineffective teaching. In any school or district 
rather substantial differences may be found between classes in 
the student entry characteristics to that class. If a particu- 
lar teacher has a substantial number of students who have re- 
tarded language development, come from homes that provide mar- 
ginal support or encouragement for academic achievement and 




5 

measure the objectives of the local curriculum does not offer 
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suffer from a cultural di sadvantagement it is very unfair to 
compare the test results of this class with another class of 
students that demonstrate normal language development and 
strong support and motivation for academic achievement. 

It is recognized that tests properly selected to assess 
the desired outcomes of instruction of the local curriculum are 
very useful for instruction. Such diagnostic use of test re- 
sults often provides specific direction for a teacher to empha- 
size particular contents and skills with that class, while 
another teacher may need to provide emphasis with' other skills 
and knowledge are most appropriate with that class of child- 
ren. The use of the same test results with both classes at the 
end of the year can not be just globally compared in terms of 
which class had the highest mean score, but should be analyzed 
in relation to the progress the students have made during the 
targeted instruction of the year. 

"Alright, i,t has been recommended that test results should be used to 
diagnose the strengths and weaknesses of what children have learned. Why 
not list the weaknesses that show up in each teacher's class and then give 
the teachers some training in teaching those things the children haven't 
learned? Parents and students agree that there are good and poor teachers 
in every school. This is just a request to train the 'poor' teachers uo 
become 'good' teachers, and even give them some incentives such as a bonus 
or a raise. If the teachers don't improve with the inservice training then 
the district should fire them or transfer them to other work." 

Several myths are behind these comments. 
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Student learning is solely related to whether they have been 
taught particular content during a particular year\ 

What and how students learn is the result of what they bring 
with them to the classroom, what their peers are doing, what 
the home does to encourage and support learning, as well as 
what is organized and presented in the instruction 



"Poor" teachers lack the knowledge of the ^uhject matter ar i 
can become "good" teachers by having more training withTTfe 
content of the subject they are teachin g. 

\~ 

Instruction has many facets that relate to the diverse charac- 
teristics of a heterogeneous group ot'students, the attitudes 
and work "habits that may or may not be developed by" the stud~ 
ents in previous education"! 



There 'is a single best way for all teachers to teach all 
students. 

Children have different styles for effective learning. Teach- 
ers also have different instructional skills and different" ~ 
styles for effective communication and motivation of students.. 
The same method of teaching and the same content will not be 
equally effective for all students in a class . 7 



Labeling "good" and "poor" teachers is an effective technique 
to improve instruction and -statr morale . 

Focus on personal inadequacies tends to produce negative or 
defensive reactions. Public comparisons of staff produces 
alienation and distrust among staff . 



An ineffective teacher can be "fired" or transferred at any 
time by the administration , — — — — — — — 

There are powerful legal and political constraints on the fir- 
ing or transferring of professional school personnel. Teacher 
contracts and contracts with the teachers' professional organi- 
zations require lengthy due process with detailed documentation 
that must be proven to be reliable and valid evidence for 
action. 
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Trie suggestions of demands for the identification of incompetent teach- 
ers and the institution of remedial action or dismissal are generally re- 
lated to the assumption that the learning of students is directly related to 
the competence or behavior of a particular individual . This assumption 
ignores the reality of the school functioning as a dynamic system that is 
influenced by the combined contributions of all the staff, administration, 
board of education and. the community. 

The purpose of citing some of the myths that surround the demands for 
immediate action to improve student learning is not to deny the constant 
need for improvement or to offer rationalizations as to why change is impos- 
sible. In contrast, these illustrations are offered as evidence of the com- 
plexity of human learning and the complex relationships that exist between 
the school, the staff, the children and the parents of the community. 

The identification of the personal and professional characteristics of 
effective teachers, is a worthy yet very complex undertaking as has been 
summarized by decades or research by such persons as Michael is, Ryan and 
Gage. Thse extensive studies of teachers. and teaching provide evidence for 
the unproductive results that may accrue from efforts to change individual 
persons or single elements of the educational process. Rather than giving 
support to the quick and parsimonious demand, it is more profitable to con- 
sider the answers to questions concerning the multiple causes of behavior 
(and learning) and the operation of the school within the content of the 

^community it serves. 

The request to identify incompetent teachers and then do something 
about them must also be considered in relation to the conviction that the 
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teacher is the key element in the facilitation of student learning. The 
foregoing discussion pf the myths and erroneous assumption regarding the use 
of a single evidence of incompetence, such as average class test scores, 
suggests that a comprehensive evaluation ofjt-he processes and product of the 
instructional program be made. The evaluation of the teacher must be made 
within the content and expectancies of the teacher in. the organizational 
system of the district. The, case studies presented in Chapter 4 suggested 
that the teacher's role (and direction or supervision) is different in the 
highly centralized system of goals, objectives, materials and methods being 
decided and managed by the central r off ice as contrasted to the system that 
relies on decision regarding conduct and sequence of instruction to be made 
by each teacher in the classroom. A different role and process of instruc- 
tion may be found in the system that charges the school principal with the 
responsibility of designing instruction ''that will be most effective for the 
students of that particular school. The tfTree organizational systems found 
in the case studies suggest that the role and expectations of teachers may 
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be quite different in the various systems. 

Recognizing the varying role and expectation' of the teacher in differ- 
ent schools and systems suggests that the criteria of teacher competence 
must relate to the local system of governance and supervision, and the 
expectancies that are held for teachers within the local system and condi- 
tions. In effect, it is essential that the school system (and the teachers) 
reach explicit agreement on the role, responsibilities and outcomes that are 
anticipated. 



A different, perception of the role, responsibility, behavior and out- 
comes of teachers is commonly held by students, parents and community. The 
non-educator -is likely to believe that the role, actions and behavior of the 
"competent teacher" is- always the same in any school or with any type of 
student. Thus, the student or parent may have an unstated image of the 
"ideal teacher" or, on the other hand, be most concerned with associating 
any undesirable student performance with some deficiency in the act of 
teaching. While the random observations of students and parents may seem _ 
inaccurate or unjust', they do present evidence of attitudes, feelings and 
expectations of what a competent teacher should be. Systematic evaluation 
is not served well by summarizing the random observations^that are re- 
ported, tin the other hand, a survey of all parents, requesting their re- 
sponse to inquiries concerning desirable and undesirable behavior of teacher 
may provide a substantial body of information for the consideration of cri- 
teria of competence. \ 

' Of course the strong contention that must be drawn from several decades 
of research. on teacher competence that is no single* criterion has been found 
to be valid. uThus, it seems imperative to approach the issue of the eval- 
uation of teacher competence' as one that will most likely produce multiple, 
criteria , of competence. The criteria must also include process and product 
if .they are to reflect the fidelity of implementing responsibilities 
assigned as well as the outcomes that are anticipated. There is no proven 
list of such multiple criteria of teacher competence but the following 
questions ana; criteria are frequently proposed. 

° Does the teacher provide each student clear and unambiguous 
explanationjif-the content being studied? 
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° Does the teacher^rovide the students with exact and 
understandable -Assignments and expected student 
achievement? 



° Does the teacher give clear explanation of how e^ch 
student's behavior and achievement will be evaluated? 

° Does the teacher demonstrate thorough understanding of the 
content and sequences of the curriculum for which the 
instruction is provided? 

° Does the teacher demonstrate acceptance and understanding of 
individual student's development, needs, and differences? 

° Does the student behavior and achievement reflect the 
teacher's ability to motivate students for constructive 
participation and high achievement? 

° Does the attendance of students reflect a positive attitude 
toward participation in the class activities? 

° Do the measures of student achievement show continuing 
progress from entry to exit from the class? 

° Do students of different levels of achievement at entry to 
the class show comparable progress throughout the duration 
of the class? 

• • • .etc. , etc* 



Such criteria as the aforementioned must be , considered within the con- 
tent of the educational system and the responsibilities that are given vari- 
ous priorities. The prioritization of needs becomes an initial step, in de- 
termining the inservice interventions that are appropriate. The development 
of recognition and understanding of the roles and responsibilities of the 
teacher as defined by the system as well as the various outcomes that are 
expected by the students, parents, and system is an essential step in 
addressing the request -- "Why don't we identify incompetent teachers and do 



something about them? 
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III. Which District Has an Instructional Program That 
Has Been Proven to Improve Test Scores? 

"We members of the Board of Education are convinced that the local cur- 
riculum and instruction are responsible for the low achievement. Some of 
the district staff have suggested that we should budget for a comprehensive 
evaluation of our program and then develop new materials and programs to 
replace what isn't effective." 

"This sounds like the same old request from the administrative staff to 
budget lots of money over several years to develbp a new program. Instead 
of wasting time and money 'reinventing the wheel,' it would be more 
efficient to purchase a program that another district has proven to be 
effective for improving student achievement. Will you tell us where we can 
purchase such a program? - " 

This statement by the school board member reflects the frustration of 
the board of education with receiving continuing requests from the staff for 
additional time and money to develop new curricula, materials and instruc- 
tional programs. The request for the identification of a program that was 
used in another district in which there were high test scores wa\ probably 
based upon some assumptions that deserve examination. The request also re- 
flects the unstated belief that there is a direct relationship between the 
books and instructional materials used and the test scores. 

^Jnon: Low student achi evement test scores are due to the same 
c c onditions in all schools and all districts . 

Fact: lo w student ach^fveUt test scores may be due to a variety of 

• Tactors . \^ 
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Discussion: Many school districts use the same text books, study guides, 
and instructional techniques with a wide range of resulting 
achievement. An important variable in student achievement is 
the entry characteristics of the student population. Such stu- 
dent characteristics as environmental encouragement, or regard 
for academic success, or level of language development are com- 
monly related to the academic proficiency that results from 
instruction. Academic progress has been found to be directly 
correlated with students' attitudes, skills and knowledge that 
have been accumulated in prior experiences both within and 
outside the school program. 

Myth : The same m aterials and instructional techniques will be equally 

effect! vein all schools J 

Fact: The " fectiveness of instructional materials and techniques is 

"Hipl Mnt on a v ariety of conditions within theTchool . 

Discussion: Instructional material or methods are often erroneously 

thought to have basic and generic power to promote student 
learning. This also reflects the belief that the teacher is 
less important for student achievement than the material or 
methods that are used. There is considerable evidence to de- 
monstrate that the materials or techniques are similar to the 
tools that may be used in any trade: there a^e wide differ- 
ences in the skill with which the tools are us^l and the speed 
and quality of the resulting product. 

This assertion also disregards the time schedules, facili- 
ties and structures that are available for the implementation 
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of instruction. In the "graded school" context there may be 
historical boundaries for the content or skills that may be 
presented to any grade or age. In a situation where high homo- 
geneity of content is presented for mastery by all students 
there is low likelihood that a wide range of materials and #. 
contents would be acceptable and effectively utilized. 

The developmental characteristics of the student popula- 
tion are critical variables in determining whether particular 
books or instructional materials will be effective for promot- 
ing learning. Student populations in various schools of a dis- 
trict may have large differences in general language develop- 
ment, study skills, motivation and general accrued academic 
abilities. Where there are wide differences in regions of a 
district or within schools there is strong likelihood that the 
textbook or materials that are producing high student achieve- 
ment in one school v;ill not be equally effective in another 
school with a different student population. 

The variable effectiveness of any and all instructional 
materials across a broad spectrum of schools was documented by 
the extensive review of reading research sponsored by N.I.E. 
This review of more than twenty-five thousand reading research 
studies concluded that no particular material or reading pro- 
gram was clearly superior to others for attaining high reading 
achievement. These findings have been reinforced by very 
similar findings of the effectiveness of demonstration heactj ng 
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programs sponsored by the California State Department of Educa- 
tion, The Right to Read Projects, and the national evaluation 
of Title I programs. The studies all point to the conclusion 
that instructional materials or strategies of instruction do 
not possess a generic quality that will insure a desired level 
of student achievement in any school or with any population. 

Myth : The same materials, techniques or instructional strategies will 

be equally ettective when used by any teache rT 

Fact: The effective use of instructional materials and strategies 

depends on many teacher-related factors . 

Discussion: Instructional materials do. not have indigenous qualities that 
assure common use and application. Moreover, the assertion 
that the critical variable is the material ignores the skills 
and knowledges that ere usually required for effective use. It 
is also apparent that the assumption suggests that learning 
results from exposure to the materials rather than the dynamic 
interaction of the student, the teacher, the instructional 
materials and activities, and the context of the facilities and 
structures of the situation. 

Experience with the often-recommended strategy for diag- 
nostic - prescriptive - individualized instruction suggests 
that particular skills of the teacher are required and that 
these skills differ from those required by the lecture method 
or inquiry strategies. Various materials may require different 
preparation, presentation, or use, with students of various 
devel opmental characteri sti cs . 
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Recognition of the different teacher-student interactive 
requirements may be observed in watching the implementation of 
such reading programs as "The Basal Series"; the Phono-Visual; 
the Sullivan; "The Language Experience Program"; etc. No mate- 
rials are "teacher proof" as some would allege. Materials and 
programs are only tools that may be employed with varying 
effectiveness and success. 

The purchase of a commercially published or district-developed 
program is less expensive than developing a new program in the 
local district . 

The initial cost of the materials and instructional strategies 
is rarely the full expense of implementing a new program . 

The comparison of the cost of purchasing a developed program 
and associated materials with the cost for developing a local 
program over a period of several years will commonly suggest 
that the purchase of an externally developed program will be 
less expensive. The case studies described in Chapter 4 illus- 
trate how these districts engaged their staffs in developments 
that took five or ten years and there is ongoing development 
arid modification. 

The critical issue in this question of expense is related 
to the often hidden costs of staff deveiopmnt in understanding 
acceptance and perfecting the skills needed to implement the 
new program. With the purchase of a program developed externa 
to the district there is the need to educate the staff on the 
purposes, contents and procedures as well as making necessary 
adjustments in structures or schedules that will allow 
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implementation of the program as it was designed. The full 
costs' of these developments for staff and program structure may 
extend over several years after the original purchase of the 
program. 

Thus, the comparison of expense of the purchase of an ex- 
ternal program with a locally developed program must be based 
upon the full range of staff involvements that are required 
until the new program is operating and being consistently 
implemented throughout the district* This statement of 
accounting for the expenses required for implementation prompts 
the observation that it is essential that the district estab- 
lish a procedure for monitoring the fidelity of implementation 
of a new program. The district is well served by a design for 
comprehensive program evaluation that will provide reliable 
information regarding the processes of instruction that are 
actually used as well as the outcomes that are measured. 

Myth : C hanging the instructional program is a simple, direct, quick 

and usually effective solution to the problem of low student 
achievement . 

Fact: There are many difficulties and obstacles to changing the 

Instructional program in a district and there are many possible 
advantages in changing programs . 

The potential advantages as well as the difficulties are 
reTaTed to the characteristics ot the community, the governing 
board of education, the administration and staff of the 
schools, and the student population . 

Discussion: The enthusiasm for changing the instructional program is usu- 
ally expressed by non-educators who assume that the materials 
and instructional techniques have been ineffective as evidenced 
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by lower test scores than are desired. While some of the 
instructional staff may voice an interest in having new mate- 
rials or new structures, there is a common tendency for in- 
structional staff to rely on the techniques and procedures- they 
have used and perceive as useful to manage the classroom and 
provide security for the day-to-day instruction. Frequently, 
th^resident staff of a school has spent considerable time and 
effort in developing procedures that are related to evaluation 
of student achievement and the sequence for instruction from 
grade to grade in the school. Such developments are commonly 
grounded in staff preferences as well as being closely aligned 
with the personal skills of the teacher. 

There is small likelihood that the purchase and adoption 
of new materials or programs by the administration of a dis- 
/ trict will result in immediate and effective use. £ven if the 
staff may agree with the need for change to bring improved stu- 
dent achievement, a substantial period of staff familiariza- 
tion, developing knowledge, and skills in the use of the new 
program will be required before full and effective implementa- 
tion will be made. 

The skill and enthusiasm of the staff in using any mate- 
rial or program is a strong predictor of the interest and suc- 
cess that students will exhibit with the use of the program. 
Students and teachers form important attitudes toward new prp- 
cedures that are often related to the demands of the changing 
roles and interactions that are required by new materials or 
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instructional strategies. The typical time required for staff 
familiarization with new procedures and changing revered 
practices which provided security lends much question to the 
assumption that an instructional program may be quickly changed 
through board adoption and administrative direction. 

Experience with district efforts to purchase and install a 
new program based upon the recommendation of testimony by ex- 
ternal authorities or other school staff or even by the single . 
administrative edict has commonly produced the feeling in the 
instructional staff that they have been ignored . Instructional 
staff citMiimonly hold strong beliefs that ^ttlere is bottf science- 
and art to teaching, and the teacher provides the interpreta- 
tion and integration that enhances student learning. Most 
teachers also nold strong convictions that they are in the best 
positiofl to make decisions regarding the materials and methods 
that will be most suitable and effective for the students in 
thei r school . 

If an external program is praised as "proven to be effec- 
tive" in the use by other districts, the local staff may be 
quick to reply that local students have different needs and 
characteristics than students ifr other districts. The local 
staff may also look for discrepancies between the new program 
content and what has been in us?, Even though the' differences 
may be small they will be used as evidence that the- new program 



will 'not meet the needs of the student and thus cause further, 
decline in test scores. 
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Any new procedure is commonly perceived as requiring 
"additional work" or "time consuming procedures" for management 
and use. The common reaction: "The new program demands so 
much paperwork and record-keeping that there is no time left 
for teaching!" Insofar as a new program requires skills and 
techniques that have not been used, the teacher may be 
threatened by the absence of techniques or activities that have 
been perceived as essential for the maintenance of discipline 
or management of the classroom. If the new program is per- 
ceived as confusing or disruptive to relations and classroom 
management that the teacher has felt to be successful, then 
there is high likelihood that the new program will not be fully 
implemented. The analyses of attempts to disseminate "model 
programs" under ESEA Title IV revealed that although a new pro- 
gram had been adopted and purchased by the. centralized system 
for planning and management of instruction, less than 50 per- 
cent of the classes received the instruction that was pre- 
scribed by the new program. In some cases teachers made an 
initial effort to use the program but as there were unsuccess- 
ful experiences they reverted to the procedures previously 
US ed. It must be concl uded that new materials and programs of 
instruction must be understood, accepted, and provide time for 
developing the skills for effective use if the new program is 
to enhance learning o.f students. 
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The potential advantages of changing the program of in- 
struction are increased as there is recognition that an educa- 
tional system is a "people system." Each school district 
serves a heterogeneous community and student population and the 
district 'Staff provide an array of accumulated talents, skills, 
and abilities. There is a great need for using a process of 
cJoption and implementation that involves the effected partici- 
pants in the assessment of 1) the characteristics of the stu- 
dent population; 2) the relative effectiveness of the curricu- 
lum and instructional technioues that have been- used; 3) the • 
relative emphasis that has been given to various contents, 
skills and outcomes of the existing curriculum; 4) the skills 
and abilities and preferences of the staff; and 6) the rele- 
vance of the test performance th?t has suggested that there is 
need to change the program to improve student s hievement. 

\ 

Myth : Adopting an external program that has been used by other 

districts having high achievement test scores will improve th e 
test scores of the district adopting the new program . ~~ 

Fact: Test scores are influenced by many factors and one factor of 

central importance is the congruence of what is measured by the 
test that Ms used in a particular school or district and the 
content of the local curriculum . ~- 

Oiscussion: The various tests that may be used to measure student achieve- 
ment have different congruence to the content, sequence, skills 
and knowledge that may be emphasized in a local program of in- 
struction. In previous discussions in this chapter illustra- 
tions were mentioned of how various national, state or local 
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tests may sample different contents or skills or may limit or 
extend the domain of knowledge that is used as evidence of 
mastery. 

It is quite common for a school district to select a 
standardized test (from a variety of such tests) that the in- 
structional staff and administration believe is most congruent 
and relevant to the content and expected outcomes of the local 
program of instruction. Changing the materials, and techniques 
of instruction may create a mis-match between what is measured 
on the tests used by the district and what is emphasized in the 
new program". Thus, changing the texts, instructional materials 
or strategies of the instructional program does not insure 
higher .est scores as reported in another district. Test 
scores may improve, decline or stay the same depending on a 
variety of factors among which the congruence of what is mea- 
sured by the test and what is emphasized in the program is o£ 
critical importance. 

y 

"After all, then ... is it cost-effective to purchase a new program 
(either commercially prepare.; or developed in another district) that has 
been reported + i have been used by districts with high achievement test 
scores?" 

There is no correct answer for all school districts and for all condi- 
tions. The best answe/ for any particular school district wil be found by 
making a comprehensive evaluation of 1) the content or environment; 

ERIC 



2) the financial resources available; 3) the beliefs and convictions of the 
leaders and opinion makers of the district; 4) the time available for staff 
involvement in development and inservice activities; 5) the time available 
prior to implementing a change; etc., etc. 

The case studies illustrate that different administrative roles, dif- 
ferent systems for making decisions and different roles of participation by 
the instructional staff may profoundly influence the nature and duration of 
change that is possible. At least two of the case studies described devel- 
opments that required five to ten years of commitment of finances,, staff 
participation and administrative priority for maintaining the effort and 
direction. Obviously, such conditions may not exist in many school dis- 
tricts; in the absense of the several essential conditions for local program 
development the answer may be -- purchase and adopt an externally developed 
program. The purchase of an externally developed program must always in- 
clude the expectation that a substantial amount of time will be required to 
develop the understanding and skills needed for successful implementation of 
the new program of instruction. 

The anticipation of immediate improvement of test scores of student 
achievement is ill-founded and will probably result in disappointment. 
Changes in student learning [ and ....performance .o may only be meaning- 
ful ly related to the instructional program as there is sufficient time..to 
make longitudinal studies of student achievement on curriculum-relevant 
tests after the program of instruction has been documented as being imple- 
mented with fidelity and consistency. 
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IV. Why Don't We Ask The Superintendent To Develop A Comprehensive 
Plan To Improve Student Achievement And Present The Plan 

And The Related Costs Next Month ? - -* 

This question often follows a series of suggestions and discussions of 
various conditions that are alleged to be responsible for unsatisfactory 
student achievement. Each suggestion is often stated as "the solution" with 
a request for additional funds. The board of education may be concerned 
that each request often refutes the importance of the other requests, and at 
the same time, wonder how the changes or additions may be incorporated into 
the existing program. Frustration with the variety of alleged solutions and 
the increased costs of the many suggestions may prompt the board members to 
propose that the superintendent develop a comprehensive plan that has a 
specific cost for each part of the plan and a total cost. 

The persons requesting. that the superintendent develop a plan and make 
a report in one month may very well have the belief that the superintendent 
* has been putting off the request for actions to improve student achieve- 
ment. Such a belief may be reflected by the citizen observation. . . "The 
superintendent just 'stonewalls' efforts to get a specific plan in opera- 
tion, and continues to have different staff members submit proposals that 
deal with different parts of the program." 

While this question/request for the superintendent to develop a 
comprehensive plan seems to be logical, it is made and based on several 
questionable assumptions* 

7.3 
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Assumption 1: The superintendent knows or has access to the technical 
"knowledge of the causes of unsatisfactory achievement of 
students as well as the interventions that will produce an 
effective and workable plan , : 

Discussion: The multiple causes of student learning or low achievement 

require a thorough analysis of both out-of -school conditions 

as well as the productivity of the existing instructional 

progpam. The request for the superintendent to develop a 

workable plan in one month makes the erroneous assumption 

that al 1 c technical knowledge is available and whatever plan 

is presented will be readily implemented within the existing 

operations of the system. 

Assumption 2: The superintendent has major responsibility for improving 
pupil achievement^ 

Discussion: While the superintendent is responsible for the management of 
all aspects of the educational system, it is erroneous to 
assume that the superintendent is directly involved in the 
day-to-day implementation of the instructional program. In 
any educational system the instructional program is delegated 
to curriculum specialists, school administrators and teach- 
ers. Moreover, the day-to-day instruction by teachers may be 
required to be varied from class to class and school to 
school in response to the needs of the particular group of 
students, The variations in school populations, sta^f 
skills, knowledge and modes of implementing the curriculum 
demonstrate the improbability of a parsimonious plan to in- 
sure improved student achievement in the short span of one 
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month. In reality, the responsibility for improving student 
achievement is shared by the entire staff who contribute im- 
portant day-to-day implementation of the curriculum and, the 
desired educational outcomes. 

Assumption 3: The first acii on shoulo be the development of a plan to 
improve "sTudent" achievement . 

Discussion: The request for an immediate development of a plan th-at will 
improve student achievement makes the erronous assumption 
that the "plan" will be universally effective with any and 
all students, and will be readily adopted and effectively 
implemented by al 1 staff within the educational system. 

Prior to the development of a plan, there is high neces- 
sity nr-eful problem identification, -in assessment of the 
needs of. ths full range of diversity o f the student popula- 
te -.r and tin?; skills of the staff as veil as the resources and 
faculties ;.hat are available within the system. 

IK- 

Jt X's vise to consider th- "Is - Should Be" model that 
prowi a documentation of vtfral currently exists (What Is) 
and tbsp makr an equally d?.U.td description of "What Should 
Be." The fairly common recast —>r improve: achieve- 
ment is a general and ambiguous request with little or no 
documentation of the ;.;i- ities that should be assigned to 
the full spectrum of student development. 

The careful documentation of the strengths and weak- 
nesses currently existing within student achievement (as well 
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as the strengths and needs of the educational system) provide 
an explicit basis for the educational staff, students, pa- 
rents avrd board of education to consider priorities that 
should t>e addressed by a workable plan. 

Tim impetuous demand for an immediate development of a 
plan 5 :- |fc ior-£o problem identification, needs assessment and 
establishment of commonly agreed upon priorities invites the 
probability of negative reactions. There is high likelihood 
that the "plan 4 will not address the various concerns of the 
publir fconcralfcd in the general request for improved student 
achieveme it< * There is equally high probability that the 
staff rr«ay view the "plan 11 as an unrealistic or impractical 
strategy in relation to what currently exists or what may be 
if»c ; emesvted within the resources and facilitites that are 
.available. The development of a plan prior to the careful 
examination of the information gathered through the "Is 
Should Be" assessment is a classic illustration of putting 
the "cart before the horse." 

As sumption 4 : Costs of a plan can be inventoried and assigned dollar values 
~ ~ easily and quickly. 

Discussion: The experience with attempts to implement a change in curri- 
culum, special resources or instructional strategies suggest 
that even very simple changes have many time consuming and 
costly "start-up" operations. 
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A prime illustration of the difficulty of assigning time 
and costs to- achieving a significant change in instruction 
is the complexity of the staff inservice required. 

There is great wisdom in the statement — it's a long 
distance from concept to construction . While it is rela- 
tively easy for one or several persons to conceptualize a 
plan for altering the instructional delivery system, it is 
very complex to achieve understanding by the staff who must 
not only understand the concept as well as the actions re- 
quired to implement the plan. It is rare that an educational 
plan can be developed in such great detail that unforeseen 
needs will not emerge in the process of staff inservice and 
organizing the implementation. Such unforeseen needs and 
requirements may cause greater time and expense than speci- 
fied by the initial plan. 

Since the plan involves the mobilization of the actions 
of staff with diverse existing operations it is highly impro- 
bable to assign discrete dollar values to the stages of a 
statement of a plan to the final implementation. 



Assumption 5: The statement of a plan for improving instruction and student 
Assumption - achieve[nent can predict and control the s u bsequent DenavToTr 
of teachers, administrators and other start . 

Discussion: The verbal description of a plan for improving student 

achievement may present an explicit series of changes that 

, sound logical and possible. However, each change Wi'»l 
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require some changes in the behavior of the teachers and 
administrators responsible for the conduct of the program. 
As a plan calls for adopting new behavior there is a concomi- 
tant requirement that previous behavior be abandoned. It is 
questionable whether any plan (quickly developed without ex- 
tensive participation by the staff) can predict how adminis- 

I 

trators and teachers will react to the plan or whether they 
will find it possible or feasible to implement within the 
other perceived demands of the school program. 

The extensive efforts of the U.S.O.E. to disseminate 
"innovative plans and projects" to many school districts met 
unforeseen obstacles. While. there was often initial accept- 
ance of the need or rationale for the plan, when the local 
staff were engaged in implementing the plan a variety of con- 
ditions and established patterns of behavior were obstacles 
Noncarrying out the plan as stated. 

.A pertinent behavioral element that may confound a com- 
mon plan for instruction is the preferred cognitive and com- 
municative style of each staff member. The statement of a 
plan that may require a particular style. such as didactic 
lectures, group discussion, student directed inquiry, etc., 
runs the risk of being alien to some individuals, or per- 
ceived as unworkable in the classroom. 
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Summary Point of View 

The request* for the Superintendent to develop a plan for the improve- 
ment of student achievement is commonly perceived by citizens or trustees as 
logical and appropriate. The request for a plan to be developed within one 
month shows insensitivi ty to the dynamics of an educational system and the 
amount of time required to conduct a meaningful needs assessment, problem 
identification, ajjp prioritizing the multiple needs that are presented. 

There is a compelling need to inform those requesting quick changes 
that the educational system involves the interactions of the. content of the 
community with the students, the staff and the existing organization. The 
presentation of the variety of needs as well as the variety of perceived 
problems should lead to a "plan for planning" rather than the quick develop- 
ment of a plan, 

i 

The discussions of some of the assumptions that are made are not inten- 
ded as rationalizations to ignore the request. In contrast, the examination 

of the assumptions suggests the compelling need to avoid' hasty response and 

i ■ 

present a plan for planning that will allow ample participation and assess- 
ment of what currently exists as well as^what should be. 

The creation of a systematic plan for the development of a plan to im- 
prove student achievement affords an opportunity to have the staff develop 
understanding of the needs that are to be addressed by a modification of the 
existing program. The introduction of new materials, structures or instruc- 
tional techniques without the understanding and acceptance of the implement- 
ing staff may produce hostility. The plan for pi anning-aVTows- discussion of 
natural resistance to change by instructional staff who may have strong con- 
viction that the methods used are appropriate. 
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Since an educational system is a "people system," the importance of 
thorough communication of the needs that are cited as requiring a change as 
well as the alternatives that are suggested cannot be minimized. The ab- 
sence of thorough communication between the administration and staff often 
results in the teachers believing that their knowledge and contributions are 
ignored. Certainly, a common reaction of the instructional staff to a quick 
edict that the instructional program will be changed is a passive attitude 
rather than full commitment to implement the new program as effectively as 
possible. 

Skillful administrative leadership encourages -planners, and supervisors 
of implementation to anticipate that some or many staff members will resist 
change. The communication with staff and community must patiently provide 
opportunity for expression of diverse points of view and then an equally 
patient but firm insistence that the alternatives must be prioritized and a 
decision for acceptance of a plan for improvement adopted. There is equal 1 
importance for common understanding that when the decision for a change is 
made there is a commitment that the plan will be utilized for a specified 
period of time to allow determination of the effectiveness of- the plan. 
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CHAPTER 4 



/ 



School District Stories 

in which six districts tell how they 
developed and implemented systems linking 
testing, evaluation and instruction to 
improve student learning . . . 
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CHAPTER 4 
School District Stories 

Introduction 

t> 

Each of the following stories reveals hard working administrators and 
teachers dedicated to helping children learn. These six districts are 
"heroic" in that they persevered for long periods of time — most for longer 
than five years — to develop a comprehensive system for monitoring student 
learning. The systems they developed, however, were not simply descrip- 
tive. The tests they administered and the evaluations that they conducted 
were intended to point to remedies, and to modifications of classroom in- 
struction- To the casual reader, the^di stricts ma^ seem very dissimilar to 
or;r another. So may the specifics of the work that was done by each of the 
central office staffs from start-up to the present. In this introduction, 
we will point out some of the similarities underlying the myriad of de- 
tails. It is these similarities which may make it possible for other dis- 
tricts to recognize their, own situation and to assess the extent to which 
they want to move in a parallel direction. 

Environmen ts. The six districts vary from small to medium size. With 
the exception of Vallejo and Cincinnati, they are primarily suburban, acade- 
mically oriented districts with a relatively stable population. They have 
been stressed in the same way as other U.S. school districts: some popula- 
tion shifts; some budget problems; some external controls in the form of 

i • ■* , 
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federal and state requirements accompanying federal and state monies. None- 
theless, when compared with big city districts, with rapidly changing condi- 
tions and large numbers of non-English speaking children, these districts 
seem to have had relatively more energy to turn to instructional matters. 
Good instruction was important in these districts, and to their parents, 
teachers, students, and administrators. The press fo .1 test s.cores — 

believed by the community to more or less accurately i ; itudent 
achievement -^-was accompanied by a district comr.iitment to make itself the 
accountable agency for student learning. N 

In "these districts, the federal and state role, not only in terms of 
providing money for special programs and imposing requirements for testing 
and evaluation, but also in stimulating new systems-oriented approaches to 
educational problems, should not be underestimated. The six districts 
represented here chafed, as did others, under paperwork burdens. Nonethe- 
less, they often responded to the intent that the written regulations may 
have masked. San Juan's commitment to school level decision making was 
influenced by Cal i forni a 's. Early Childhood Program. Vallejo's interest in 
testing and its relation to instruction was stimulated by the California 
Assessment Programs: Cincinnati developed many of its needs assessment 
procedures and training' outlines for local teams through federal funds. 
Newport-Mesa's work in their Objectives-Based Reading Program was encouraged 
by UCLA faculty who did much of their research with federal funds. 

Initial district conditions , In each of the districts, you will be 
able to identify an "idea champion" and a stable core of staff. The idea 
champion or champions are individuals who were in possession of the care/ 
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clout factor. They were in a position — not necessarily the top. admini- 
strative post where their caring could reverberate turoughout the sys- 
tern. They had the clout — whether formal or informal -- to translate their 
ideas into action* They had management skills. They could bring along, 
perhaps slowly at first, others with them. 

The districts in which these idea champions worked supported their 
efforts in a number of w^ys. Persons with technical skills in developing or 
analyzing tests were either hireu or ^ ocurH as consultants Academic con- 
sultants were often brought in to stimulate new thinking or work on a speci- 
fic problem. Computer time was available from somewhere. Released time for 
teachers or summer salaries came out of district support, or was obtained 
from special project money. 

Not only were there skilled leaders, willing followers, needed techni- 
cal and managerial resources; there was also a vision of the possible and 
staying power to move in a direction in spite of setbacks. All the dis- 
trict stories describe false starts, mistakes, the need to start over. Los 
Alamitos, for one, purcnased at least two testing systems before they moved 
to writing a curriculum scope and sequence. The Evaluation/Planner in San 
Juan felt isolated and lonely. He was a voice in the wilderness for a long 
time before gradually acquiring allies. Teacher resistance to his work, and 
often resistance from others in the central office was initially very high. 
But in all the districts, leaders "kept their confidence in the testing, . 
evaluation, instruction relationship intact even while modifying tht speci- 
fics of what they were doing;x * 
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Developmental sequence . None of the districts describes a planning 
process that was formulated at the beginning and served as a framework 
throughput the development cycle. Rather, things seemed to emerge, evolve, 
and get made up as the districts went along. One thing led to another. 

Each district began its process at a different place. Newport-Mesa 
first examined what the community expected from the school districts. Los 
Alamitos began by purchasing tests. San Juan wanted to encourage and em- 
power school site planners. Vallejo and Clark County both reacted to low 
test scores by trying to express a district philosophy and direction. 
Cincinnati reacted to negative community opinion and slipping budgets by 
encouraging neighborhood-business-school relationships. 

Each district proceeded, also, through a series of stages, coordinating 
their operations in different ways as they went. Newport-Mesa and Los 
Alamitos now have a testing system referenced to objectives contained in a 
district-wide scope and sequence. Los Alamitos, especially, has tied this 
framework to a particular diagnostic/prescriptive teaching strategy, sup- 
ported by indexed media- and materials, ^-eiark County also mandates parti- 
cular teaching elements in the classroom and couples this with emphasis on 
principal supervision of teachers and central office supervision of 
teachers. 

San Juan and Cincinnati r on the other hand, are decentralized in the - 
use of data from tests and surveys. Both collect information about parer. •• , 
student, teacher perceptions and organize them into easy-to-read package, 
for school groups to use. In both these districts, the central offices 
provide training support and guidance for local teams who work through an 

- as : 
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annual proces to allocate school resources to meet high priority school 
needs. Vallejo is doing something on a number of fronts. Having started 
with school-site analysis of instruction in relation to the CAP tests, they 
moved to strong staff development programs, and finally to developing a 
district-wide scope and sequence. 

Role of major actors . The district central offices have a strong role 
to play, but the role differs somewhat from district to district. In all 
districts, a variety of tests are required. District staff analyzes or has 
analyzed, the tests. District offices provide staff development to teach- 
ers; or in the case of San Juan and Cincinnati, leadership training to citi- 
zens. District offices also provide support or guidance to teachers in dif- 
ferent ways. In Los Alamitos, texts and other materials are cross-refer-' 
enced to the tests. A learning specialist works with teachers in classroom 
management problems. In Clark County, the entire supervision and management 
system is set up to ensure accountability for clearly defined roles and 

responsibilities. ^ 
The teachers 1 role in development of the process differed from district 
to district. However, it seems that, in all districts, there was initial 
resistance, followed by gradual commitment as teachers themselves worked on 
the system. Nonetheless, it is not clear from the accounts whether all 
teachers now believe, once the system is in place, that instructional- 
improvement commensurate with central office efforts has resulted. 

Impact . The district stories imply more than they prove. With the 
exception of Clark County, there is little evidence presented of a dramatic 
increase in student scores. This, however, may not be the only evidence of 

r 
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impact that we should consider. It does seem clear that, for many dis- 
tricts, the^ r pub'ic image has improved. Los Alamitos, Clark County, San 
Juan, among other', clearly enjoy reputations as forward-looking, well- 
managed, self-renewing organizations. Teachers in all six districts as well 
as administrators may have a fee.ling of professional well-being different 
from that in most districts. The districts "al i have a clear sense of pur- 
pose and of priorities, v 

\ [More to come.] 
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LOS ALAMITO: 1 EXPERIENCE: 
ITS CRIT~^I ON-REFERENCED TESTING SYSTEM 

by David Hatton 

• The Los Alamitos Unified Sc^, District is ? small school district now 
consisting of five elementary school. • middle schools, and one high 
school . 

Our distri i. l ad its beginning loo;. _s a one-room school with an 
enrollment of 43 s . ; ots and a first ^ r budget of $700. It remained a. 

rural one-school qu..-^l until the housing boom cf the mid-1950's, when our 

•i 
i 

name was changes to Los .Vamitos School District to provide community iden- 
tity. The district continued to £jrov; until it reached a maximum of seven 
elementary schools with ar: enrollment of 4*000 in the early 1970's. As has 
been the case in many districts,, declining enrollment since then 'has re- 
sulted in the closing of two schools. 

In June of 1979 residents of the Los Alamitos School District voted to 
form a unified K-12 district which became effective in July, 1979. The ele- 
mentary district, therefore, ce&s^d to exist on June 30, ib<?0» concluding 99 
years o + * operation. The^JJnified District, with an .enrollment of 5,200, is 
comprised of the former Los Alamitos Elementary District and a portion^of 
the Anaheim Union High- School Distri,: (Oak sod Pine Junior High Schools, 
and Los Alamitos High School). 

The Los Alamitos Unified School District is an up^r-middl. class sub- 
urban setting. The large percentage of. parents who are college graduates 
have very high educational expectancies for t!:or children. As a result, 
the district has been a leader in the development of innovative educational 
programs, , and has consistently scored a mg the highest in the State on 



achievement tests, while operating at or below the State's average per-pupij 
expenditure. 

Because our parents emphasize, the importance of academics, there has 
been a continual demand from the community for student progress reporting. 
Our parents want to 3 ee consistent student progress; progress which they 
expect to be documented in a clear and understandable fashion. 

Development of the Testing System 

Formative Years 

In 1970, the district attempted to meet the need for student process 
reporting by purchasing test items to measure a sequence of behavioral ob- 
jectives. The objectives aimed out to b'e improperly sequenced, Teachers 
hated them and eventually distriuv. administration discarded them. 

" But there remained th: need for stud-.nt progress reporting. However,, 
an equally compelling neea began to emerge — individualization of instruc- 
tion. Though the behavioral objectives system had- not worked, district ad. 
ministration realized tNt r. properly -developed test could not only satisfy 
the parent demand for progress reporting, but also be a vehicle for provid- 
ing instruction appropriate to a student's current level of achievement. , 
Whole-class instruction could be replaced by smai' 5 rou? instruction. 
Students could be grouped s cording to t;.3ir performance on the test and 
instructed according to their current performance levels. 

Rather than purchase an available testing system, district administra- 
tion formed teacher committees to write their own tests. The feeling was 
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that tests written by district's teachers would be better received by other 
teachers than tests developed by an outside agency. 

In 1973, the first district-developed tests, known as PAL (Pupil 
Assessment Lab), were administered to students. They were intended to moni- 
tor student learning so as to feed back information to teachers about how 
individualized instruction was working. They also were seen as a way of 
reassuring parents who were concerned about their children's progress. 

Teacher reaction to PAL was extremely negative. Everyone complained. 

Teachers today remember their complaints: 

° It was a waste of time. It didn't tell us anything we needed to 
know. 

° The children were confused by the tests. 
° We didn't know what to do with the results. 

° We didn't want to be judged on the basis of what our children did on 
those tests. 

° It was not coordinated with anything we taught. 
Teachers made their complaints known to parents and to the Board. It was a 
hard time for central office staff. We wanted the testing system to work. 

We appointed teacher committees to try to revise the items. We re- 
sponded to the need for coordinating testing and curriculum by beginning to 
work on a district level instructional continuum, A group of volunteer 
teachers were paid to work on a reading continuum during the su^ner. Thei^r 
work continued through the school year. 

By 1976, teacher committees had generated a reading continuum (RIC) and 
the beginning of a math continuum (MIC). By 1976; the PAL criterion- 
, referenced testing system had been scrapped. Teacher committees had written 

so 
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reading, math and language arts test items to form their own Criterion- 
Referenced Tests ( r °.T). The tests were developed by the district's teachers 
and designed to support d: strict-adopted instructional continua. A more 
detailed description of our CRT's comes in a later section. 
Staff Development Program 

Realizing that a testing program by itself could not improve instruc- 
tional practices, we at the central office instituted a staff development 
program in the early 1970's. Convinced that a staff development program was 
at the very core of a support system for the criterion-referenced testing 
program, we hired a full-time staff development coordinator in 1976. Our 
program was designed to provide teachers with the skills necessary to act 
upon the results of the district-developed testing system. 

Currently, the staff development program includes two levels, of in- 
struction. Level one is based upon the Madeline Hunter orientation to 
teaching. Nearly all teachers participate in level one.. A modified version 
is provided for substitutes and aides. Covered in level one are teaching 
techniques, including task analysis, diagnosis, prescription, instruction 
and evalu^"ion. Teachers attend the staff development classes on release 
time and tnen practic... the techniques in their own classroom. Follow-up 
visits are made by the staff development coordinator. 

Level two programs are of two kinds. The first kind is an extension of 
the level one activities with attention to individualizing instruction by* 
choosing among alternatives. Additional sessions cover matters such as 
diagnosing and developing teaching strategies for meeting affective needs, 
u ng inquiry skills techniques, and teaching specific curriculum content 
such as reading, writing, or mathematics. 
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The Learning Specialist 

At the same time, the staff development program was placed in opera- 
tion. We in the district office decided to give the responsibility for 
coordinating the newly developed testing system to a learning specialist 
assigned to each of the schools. We regard the learning specialist as a 

superteacher, not as an administrator. Approximately -,-u percent of the time 

i 

of the learning specialist is spent working directly with students. The re- 

i 
i 

maining time is devoted to a range ofj other activities. The learning spe- 
cialist works with teachers in their [classrooms to provide instruction meet- 

i 

ing the needs of students as indicated by the test results and by teacher 
observation. The learning specialist sometimes conducts demonstration les- 
sons and makes suggestions for new materials. The learning specialist meetv 
with teachers when the test scores come back and helps plan for the next 
several months. A small portion of the learning specialist's time is de- 
voted to in-school inservice activities. The learning specialist is also 
responsible for diagnosing new students a^/thT^y come into the classrox^ and 
facilitating the placement of children in instructional^ groups. 
Media Center v 

Within every school in the district is a materials and media center 
staffed by a full or part-time specialist. The district essentially orders 
materials that teachers and learning specialists request. The central 
office coordinator of materials each year asks the teachers what they want, 
looks at areas in which materials need to be supplemented, and meets with 
the media specialist to facilitate the acquisition of equipment and supple- 
mentary curriculum materials. The learning specialist in each school often 
works closely with the media specialist to try out new materials as they 

S3 
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arise to determine how appropriate' they are for particular types of child- 
ren. The learning specialist may try out sample materials sent to the 
schools to decide whether they are suitable for ordering. The media specia- 
list in each school often functions in an advisory capacity to the learning 
specialist and to the teachers, recommending books, films, adjunct materials 
which he/she thinks appropriate. 

Description of the CRT 

Originally developed solely as a measure of student progress, the tests 
initiated in the early 70's have evolved into a system which assures a link- 
age between instruction and test results. Supported by the staff develop- 
ment program and by other support personnel such as the learning specialist 
and the media specialist, the system currently is in full operation in 
grades K-8. Highlights of our testing, system: 

° All tests are directly linked to a kindergarten through eighth grade 
instructional continuum for both reading/language arts and mathe- 
matics. 

o All tests are computer-scored. Results are formatted in easy-to- 
read computer reports to teachers, parents, and site and district 
administrators. j 

° Information about the testing system and how it works is. taught as a 
part of "the staff development progVam. 

° The tests are reviewed on an ongoinn basis' with revisions occurring 
four times over the last several years. 

° The testing program has been recognized by state and national teach- 
ers, administrators and college educators as an exempl ary .-model 
appropriate for many school systems. 

The costs of implementing a system like'the one currently operating 

within Los Alamitos Unified School District are high, both in terms of 
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dollars and in terms of teacher and administrator time and effort. But the 
benefits are also great. Teachers can use the tests to group students 
according to instructional needs. Teachers and site administrators can use 
summarized test results to identify school-wide instructional strengths and 
weaknesses. This information can be used to set goals for the coming school 
year. District administrators can use the results in the same fashion to 
identify appropriate instruction program priorities for both the immediate 
and distant future. Finally, the tests can be used to meet continuing man- 
dates from the state and federal .government for the setting of local gradua- 
tion standards. 
Test Uses 

Perhaps the most important use of the testing information is made by 
the classroom teacher in planning for the instruction of his/her students. 
The test is referenced to a graded sequence of instructional continua for 
reading/language arts and math, thus enabling the teacher to determine 
exactly what skills have been mastered by each student and wh'ich still need 
to be learned. An example may serve to illustrate this important nse of the 
test. Assuming a first grade teacher plans to teach calendar skills to 
his/her class. The teacher would first consult the district instructional 
sequence for teaching calendar skills. A section of the sequence is repro- 
duced below. 
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TasK Analysis * ^ Grade Level 

1. States the days of the week in order K-l 
*2. Identifies the number of days in a week K-l 

3. Locates the names of the days of the week 

on a calendar 1 

4. Reads numbers to 31 1 

5. Locates a given number on a calendar 1 

*6. Identifies days of the week for a given date f 1-2 

7. Identifies the number of dates in a month 

for a given day of the week 1-2 

*8. Identifies the dates of a given day of the 
week 

The learnings which have asterisks are known as benchmarks and are tested. 
After becoming familiar with this task analysis, the teacher administers 
benchmark tests as appropriate for each student. Generally the tests are 

i 

given to stiall groups of children. Individual performance on the test will 
enable the teacher to reorganize student groupings according to level of 
mastery of skills along the continuum. Not only can the teacher group the 
students for instruction based on their current level of achievement, but 
the teacher can also project student growth over the course of the school 
year. Those students not able yet to identify the number of days in the 
week might be projected to locate a given number on a calendar by the end of 
the school year. But a student who already is able to identify the dates of 
a given day of the week might be project od to master one of the learnings 
that falls at the second or third grade level of the task analysis. 
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It's important to note that the testing is designed as a gros's diagno- 
sis of Student performance. After teachers review the test results, they 
decide whether to move back or forward along the continuum and develop their 
own form of assessment as they proceed along the instructional sequence. 

In summary, individual teachers use, the test for gross diagnosis of 
student's current level of performance and for projecting and measuring 
student growth qrver the course of a school year. Teachers as a group also 
work with school principals to identify instructional priorities according 
to student performance on the tests. Since the tests measure several con- 
tent areas within reading, language arts and mathematics, it is possible to 
identify student strengths and weaknesses within each of these disciplines. 
As an example,- analysis of global performance in reading might reveal that 
as a group, seventh graders had strength in identifying the main idea of a 
short story, but that they had difficulty in predicting the outcome of the 
story. The teachers and their principals might identify "predicting out- 
comes 11 as a high priority instructional area for the coming school year- 

The testing system also allows teachers and administrators to determine 
the percentage cf students performing below, at, and above grade level. 
This information may be used for a variety of purposes, including the vali- 
dation of other standardized tests mandated by federal and state regula- 
tions. 

Another use of the testing information occurs at the district level. 
District administrators can review test results with site administrators to 
set district and site-leveV instructional priorities. 

Finally, the testing system can be used to meet proficiency standard 
requirements currently enacted in some states. California State law 



requires all school districts to establish high school graduation standards 
and to provide a warning system beginning in fourth grade for those students 
not making satisfactory progress toward graduation. The testing system 
developed in Los Alamitos fits in" nicely with these requirements, Los 
Alamitos students who are found to be two years below grade level in any of 
the tested areas ; are identified as needing additional remedial assistance. 
In this way, the system is used not only to g'uiOf instruction for. all stu- 
dents, but also to identify those students who nee'! remedial assistance. 

Factors Contributing to Successful Implementation 

The system sounds attractive. But I've already indicated that it was 
costly to develop, and that may make it not feasible for all schopl organi- 
zations. A number of factors expedited the implementation of our system in 
Los Alamitos: 

/ 

Consistent direction from leadership . Though school board composition 
and superintendents changed over the last fifteen years, commitment on the 
part of these leaders was strong. Since the system is expensive, support 
from leadership within the district must be strong and consistent. 

Principals as instructional leaders . Principals are viewed as and are 
trained to be, the instructional leaders at their schools. Because of this, 
administrators are knowledgeable with regard to curriculum and instruction 
in each classroom. Teachers and parents know that principals 1 administra- 
tive responsibility is not limited to public relations, scheduling and dis- 
cipline. Principals knort about instruction and can communicate knowledge- 
ably with teachers about the learning of students within each classroom. 



Staff development . Our commitment to^ ongoing staff development has 
expedited the implementation of the testing system. Through staff develop-' 

ment programs, teachers learn about the tests but more importantly about the 

? 

instructional continue the tests are designate support. Teachers not only 
learn how to diagnose appropriate learnings for a particular instructional 
•group but also experience new and effective instructional techniques. The 
emphasis' in the staff development program is upon continuing education for 

teachers, pur focus is not upon teaching teachers to test. Rather, it*is 

} 

upon teaching teachers to teach. Testing becomes a minor though essential 
part of instruction. 

The widespread participation by teachers in the staff development pro- 
gram has had the .consequence of providing teachers with methods of acting . 
upon the results of the district-developed.tests. Without this instruction, 
teachers would not have the tools to effectively use the results of the 
tests and non-use or inappropriate use would follow. £ v 

Adequate funding . The (development of the tesfing system was funded at 
an adequate level by our district. We also recognized from the beginning 
that implementation of the system would be expensive. Not only *was appro- 
priate computing equipment purchased to implement the system, but well- 
trained personnel were also employed to operate the equipment. 

The Learning Specialist . In order to successful ly operate a testing 
system similar to the one oper^ing "in Los Alamitos, someone must serve in 
the role of learning specialist at each school site. The learning special- 
ist is a resource to teachers ami assists the teacher in acting upon. pre- 
scriptions implied by the student's test responses. The learning specialist 
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brings the teachers suggestions and possibilities for instructional alterna- 
tives^'more Importantly, he or she provides extra instructional time for 
students who need it. The services provided by the learning specialist are 
facilitative. He/she is seen as a resource for teacher, not as an intruder 
in the classroom. 

Recruitment and selection . The final, yet the most important component 
to the successful implementation of f a testing system, is the expertise of 

.'the teachers within the district. The district has long held to a policy of 
rigorous personnel selection. The selection process is comprehensive aRd 

1 requires candidates to demonstrate their teaching skills prior to -being se- 

« 

lected. As a result of this rigorous selection process, teachers within the 
' district may be generally characterized as enthusiastic and committed to 
teaching. They set high expectations for themselves and for their stu- 
dents. The district's teachers definitely are its most important resource. 

Mistakes To Be Avoided 

» — , 

The road to full implementation of our testing/instructional program 
was not easy. Mistakes were made along the way.. A few that would seem to 
apply to. others i nterested. embarki ng upon a similar, but hopefully less 
rocky path, are as follows: 

D on't let^test development precede curriculum development . ■ In 1973, 
when PAL was developed, Los Alamitos teachers and administrators selected 
"performance outcomes" and then wrote test items to support them. The 
performance outcomes were not selected .from an .instructional sequence and 
therefore were" meaningless in terms of informing the teacher where a student 
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had been and where he/she now needed to go. The test based solely on 
t 

performance outcomes became something to be passed laid then forgotten. 
Much likkthe High school proficiency exams, which are requtreu in some 
states, mu*h effort went into preparing students to pass the exam, but once - 
passed, teachers had no direction or incentive to move^to higher levels. 
It's important to note that not all of the learnings in our instructional 
continuum are selected for the test. If the test is failed, the teacher 
knows he/she must move back along the continuum to find the point at which 

to instruct the student. Passage of a test on a learning item means the 

* ■ 

teacher can look to the next step in the instructional sequence. Obviously, 
the instructional sequence must have been carefully developed and be subject 
to revision by teachers. - 

Don't overemphasize test results . Tests are readily over-interpreted 
by teachers, administrators and parents. Strangely, those teachers who seem 
to be most threatened by tests, especially tests given to every student in 
the district, are the same teachers most apt to oyer-rely on test results as 

A 

measures of student progress. While threatening to them, test results seem 
to offer a certain sense of security to anxious teachers. A teacher lacking 
confidence in his/her teaching skills may emphasize test performance as a 
substitute for good teaching. We know that good test results may not always 
indicate good teaching practices. Conversely, good instruction always leads 
to good test results. 

Administrators and parents sometimes contribute to the overemphasis on 
test results. Tests have an "objective" aura about them. However, if re- 
sults are not combined with such powerful subjective measures as observation 

f 

r 

100 



of classroom climate, student o?al work, and student interviews, test re- 
sults are prone to inflation. A balanced evaluation of student progress by 
parents and administrators as well as teachers is^essential to the effective 
use of a testing program. At least annually, our administrative staff meets 
as a group to discuss strategies to avoid overemphasis upon testing. Gen- 
erally, such strategies include highlighting other measures of overall stu- 
dent performance which indicate strengths and weaknesses in a particular 
school's instructional programming. Such brainstorming by administrators 
and teachers of alternative measures of evaluating^nstructional programs 
has been quite productive. > 

Just as we must reassure teachers who r are resistant to testing that the 
test was designed 'as a support to instruction, so must we encourage teachers 
who are overly test conscious to use test results as only ope measure of 
student progress. In the latter case, teachers may become so preoccupied 
with "good" test results that students are^ushed to higher levels measured 
by the test at the expense of broadening or enrichment of skills at a lower 
grade, level that are important but not tested. The very best teachers are 
often trapped by this problem, especially because of the pressure for high 
test scores from our parent community. 

The school principal is one person largely responsible for controlling 
the overextension of testing. Unless the principal reinforces other mea- 
sures of student progress, students Will be pushed to higher academic levels 
without receiving important, supplementary instruction. Striking the balance, 
between broadening of current skills and extension to higher. level skills is- 
an important responsibility of each site administrator. 

*4 
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Don't rely on outside agencies to develop your testing system . In 
1970, our district contracted with a testing company to develop a test and 
to process the testing information'. While the test items were good, the 
data processing was slow as is often the case when an outside agency is in- 
volved. The turn-around of testing information was so slow as to make the 
results meaningless to our classroom teachers. 

But even if the data processing had been good, there still would have 
remained a need for teacher "ownership" of the test. Such ownership comes 
about only when teachers within the district have actually participated in 
the test development process. While it probably is more costly to take the 
time to train teachers in item writing skills and to do your own statistical 
item analysis, the long-ten? benefits gained ffom teacher identification, 
with the test will go a long way to insure successful implementation of the 
test. 

Questions Commonly Asked of Us 

-K. V * 

Throughout the years of implementation of the district's testing pro- 
gram, ^ number of questions/conceVns have arisen again and again. The most 
< salient are summarized in response to the following questions: 
1. Where does teacher accountability fit? 

T^y as we in the central offic£ may to avoid the issue of teacher 
accountability and test results^ the issue comes up every time a new teacher 
learns about the test or an experienced teacher complains about overemphasis 
of testing. This issue should not be avoided. In^fact, it should be 
pointed out that an overemphasis on accountability leads to inflated and 
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meaningless test results.. Certainly, accountability as to be a part of a 
testing program.* Indeed, the source of the pressure for accountability 
should reside with a teacher's desire to move his/her students toward com- 
fortable and attainable goals, V\s stated earlier, overemphasis on test 
results by teachers, parents, or administrators usually leads to inflated 
test results. A tool which can be extremely helpful to instructional 
improvement is thereby rendered meaningless. 

2. Would you advise developing our own system or purchasing someone el se's? 

Develop your own system, if possible. The advantage of having those 
who have to implement the system, namely the teachers, identify with the 
system, far outweigh the advantages of developing the system outside the 
district or purchasing another system developed by someone else. Even if / 
some .of. the technical aspects of test development have to be sacrificed, it 
is much better to have a system in whic^i teachers are confident. 

3. How about staff development? 

This program is essential. Without it, all emphasis .will be upon test- 
ing, rather than teasWng. Again, the object of a good testing system is 
not good test results in and of themselves. The object is to support and 
supplement good teaching practices. An inval uabl e -aid to emphasis upon in- 
struction rather than testing is the presence of a staff development pro- 
gram. Teachers must be given the opportunity to refresh and update their 
teaching skills if good instruction is to be offered within their class- 
rooms. 

4. How much will it cost if we do our, ov^n data processing ? 

A lot! Our computer programming costs alone have been $30,000 over the 
past seven years. The equipment to operate the system has cost more, though 
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the computing equipment is used for a variety of other purposes, including 
business management, and student attendance and grading," Sufficient teacher 
involvement is costly too* Teacher committees met once monthly for six 
hours a day over a two-year period to develop the reading/language arts and 
math tests currently used in the district, 

5, How do you think testing should be implemented? 

Slowly! Many districts have-made the mistake of moving too quickly 
with the implementation of a testing sytem. Simply put, the users of the 
tests — students, teachers, and administrators — weren't ready. Without 
the involvement of these groups, the tests are doomed to failure. Start 
with a needs assessment of the communities the test will serve. If only one 
school is interested, implement the system on a trial basis there. Most im- 
portantly, make sure the school board is .commi tted to a venture which will 
be time-consuming a/id costly, but which will lead to the rich benefits of 
improved student progress. 

6. What differences occur at the elementary and secondary level ? 

As most educators know, the orientation of elementary and secondary 
teachers differs' markedly. Typically, secondary teachers see as many as 150 
students a day. Students are seen in groups of 30 for slightly less than an 
hour. Elementary teachers generally work with 30 students for the entire 
school day. The large numbers and relatively small amounts of time with 
students tends to make secondary teachers resistant' to .any testing system 

\<* 

requiring extensive amounts of time to administer and score. Since the sys- 
tem described in this paper does require a significant commitment of time on 
the part of the .secondary teacher, release time was given each high school 

v. 
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teacher for the purpose of scoring the tests artd interpreting the results 
with the learning specialist. This released time paid rich dividends in 
increasing teacher nnorale towards the new testing system. 

As our CRT system was expanded to the seventh and eighth grades, the 
question of standardization of instruction arose. T^his was not a new 
issue. A similar concern was expressed when we developed our instructional 
conttnua at the elementary grades. The concern stems from the fact that it 
is necessary to have a standard curriculum if a standard test is to be de- 
veloped. Sojrie teachers indicated that their freedom to select materials and 
course content was threatened by the need for such a standardized con- 
tinuum. Our response to al 1 grade levels has been that the purpose of a 
standardized continuum is to assist student learning of skills that have 
been deemed important by teachers in the district. The continua were devel- 
oped by teachers Tor teachers; the intent was to assist instruction rather 
than' control it. 

In no way is the "how 11 of instruction dictated by the continuum. * 
Teachers are given full responsibility for selecting and implementing . 
instructional techniques to teach the skills listed in the district 
continuum. 

Summary 

. * 

The system described in this paper has the support of the teachers, 
parents, and administrators in the Los AlamVtos Community. Its greatest 
asset is that it was developed by the district rather than by some outside 
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agency. It works because test results are not overemphasized and because 
teachers who use the system are provided with the training necessary to use 
it effectively. Prc/ision of proper training has meant the addition of 
support personnel in the district. The bottom line is that having a system 
similar to the one described in this paper is much more expensive than other 
testing alternatives. However* the expense is far outweighed by the 
excellence of instruction that is made possible by such a system* School 
entities -who find themselves in a situation similar to that of Los Alamitos 
and who can afford the cost of a district -developed system should certainly 
consider this alternative. 
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THE NEWPORT-MESA EXPERIENCE: THE HISTORY 
OF A COMMITMENT TO CBE 

by Dale Wool ley 
and Nola Rochelle 

Many "old-timers" in this district recall an incident that occurred to 
Don Hout. Nearly fifteen years ago, Don was an elementary principal who 
was paid a visit by an interested father during the summer. The father was 
enrolling his 1 fourth grade youngster for the fall semester and was inter- 
ested in helping his son make a successful transition from his former school 
district to Newport-Mesa. He asked Don, "What is it you expect a typical 
fourth grade student in your school district to know when he or she enters 
fourth grade? What skills do you expect him/her to have already mastered? 1 ' . 
Don was concerned that this very reasonable question was difficult, if not 
impossible to answer. He could talk in terms of curricular materials to be 
studied, but not specific instructional goals to be mastered. This type of 
inquiry from the middle to upper income parents that characterize the 
Newport -Mesa District leads one to understand why in 1965 the Board of 
Education selected Inland Newcomer for its superintendent. 

Lee Newcomelh met the Board of Education's criteria for a new leader. 
The board was interested in finding out what the main educational priorities 
of a school district should be. Lee Newcomer was schooled in accountability 
and suggested a data gathering system that would provide them with just such 
information. To accomplish his mission, the new superintendent brought on 
board a new director of research and development from his native Clark 

County, Nevada, "the new driving force-was to be Les Shuck, an administrator 

\ « 

who would provide a detailed blueprint for\ a CBE system that would guide 
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Newport-Mesa for ten years* For most of this time the district would be 
steered by three men — Lee Necomer, Norman Loats (then Associate Superin- 
tendent), and Les Shuck -- whose joint philosophy it was that the district 
should be in control of the "what" of education (the important skills to be 
learned), while the schools would be in charge of the "how." For many 
years, -the district was to be decentralized. Each school would have its own 
budget and the freedom to achieve the "what" ir whatever way the principal 
and staff saw fit. 

The project which initiated Newport-Mesa's question for the "what" of 
education was called the Instructional Tasks Project, The I.T.P. was writ- 
ten by Dr. Shuck with the help of some educational consultants and funded by 
ESEA Title III or federal monies. Its goal was zo determine the contribu- 
tions that the community expected from the school district. In process, it 
involved in-depth interviews with a stratified random sample of community 
members. Participants were asked to describe specific incidents from their 
past where the behavior of young people served as good or bad examples of 
the kinds of skills and knowledge they thought young people should have. 
The results of this one and one-half year study included the finding that 
some skills and knowledge were considered more important than others*. 
Exactly what primary and secondary responsibilities the schools should 
assume became the Board's most important policy. It was adopted in 1970 and 

entitled the "Statement of Educational Principles, 11 thereafter known as SEP 

i 

for short. 

SEP dictated that the^schools' primary responsibility was to help stu- 
dents develop their maximum intellectual capacity in twelve specific skill 
and knowledge areas. These twelve areas included: 
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reading computation fine and practical arts 

composi ti on mathemati cal systems doi ng 

a listening science thinking 

speaking social studies^. language systems 

Since reading is always an area of great concern, the district began with 
this subject area in its first attempt to further specify the skills com- 
prising this general skill. As it would on many, occasions, the district 
would turn to university consultants and research and development centers 
for suggestions as to how to proceed. The project of reading led to OBER . 

Although the devel opment _of behavioral objectives was becoming quite^a 
popular thing to do* in many education circles, the district had a hard time 
finding examples where an entire curricular area .had been translated into a 
continuum of specific instructional .outcomes. The new director of the dis- 

I 

trict's Development Lab, Bob Otto, hoped to adopt or adapt the work that 
had been accomplished outside the district to the district's needs. Otto 
found that Professors James Popham and Rod Skager were devel oping -a^con- 
tinuum of instructional objectives Jn reading, grades 1-6. Jtewport-Mesa 
then entered a joint venture with U.C.L.A's Center for the Study of Evalua- 
tion, where ths continuum was being put together, that allowed the district 
to use the objectives and test items.*, This continuum was to be'called OBER 
far Objective-based Eval uation--Readi ng and was to represent not just the 
objectives that; were important for 5 every student to master, but rather al 1 
the objectives! that could be used to define what was meant by "reading" at 
the elementary school level. Interested teachers in the Newport-Mesa 
schools were asked to review the objectives and choose those they thought 
important for students to master at various grade levels. Most partici- 
pating teachers did this on school time and for no extra pay. Objectives 
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chosen by participants from more than half of the schools were then used to 
conduct district-wide assessments. Otto and his group could then paint a 
picture of how students were performing on these chosen objectives. The 

r . 

results of this assessment yielded more than the usual data. 

One unexpected piece of information was not the least academic in • 
nature. It was learned that school staff members have a strong tendency to 
be unconcerned about the process of choosing instructional outcomes until 
assessment begins to take place in their classrooms^ The OBER assessments 
generated teacher and administrator comments that the district was trying to 
"mechanize" the creative process of education. Others said that "you just 
can't measure what is taught in nr£ 'classroom (or school) in specific in- 
structional outcomes!" Despite such commentary, Dr. Shuck, Dr. Loats, and 
the new superintendent, Bill Cunningham, forged "ahead, using OBER as a model 
and learning its lessons on staff development. 

Since no other complete system such as OBER was available, the district 
set out on its own to create needed continuums in other subject areas. With 
some consultant assistance -- notably Dick Harsh from E.T.S. and Dr. John 
McNeil from U.C.L:A — and the district coordination of Cora Schultz, 
« newly-appointed resource teacher for the Development Lab, the district set 
up twelve teacher committees. There was* one committee for each subject area 
in the SEPOist or priority subject areas. ^Each committee was charged with 
the duty of developing a continuum of objectives. Through a competitive 
interview process a capable teacher was selected as chairperson for each 
committee. S/he had the freedom to select the rest of the committee mem- 
bers. Each was paid for 22 extra days of work per year. Frequently, the 
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chiivpersoQ and his/her committee members were provided with substitutes for 
all -day organizational or work meetings. Each committee had a small budget 
in order to bring in outside consultants to help them when necessary. 

As the SEP committees began to work writing behavior objectives to 
define each' subject area, twelve more, commi ttees were created to develop 
test items to match the objectives being developed by the SEP groups. These 
committees were concerned with "program outcome evaluation" and were known 
as the POE committees. Interaction' between the SEP and POE groups resulted 
in twelve continuums of behavioral objectives known as "universes" — tomes 
that were to a fledgling CBE program what the English dictionary is to 'a 
foreign-born speaker of our language. This interaction also resulted in 
banks of items correlated to these objectives, and to many heated discus- 
sions between the two committees about the merits of certain objectives. 
The writing of test items, it was discovered, caused the rejection of many 
an objective as unclear or impossible to measure. Test items .served to test 
the objectives themselves as we went along. 

At this point Research and Development and Development Lab staffs were 
beginning to anticipate some district-wide assessments. Items were available 
and the time was drawing nigh. The missing link was the need for a wider 
teacher exposure to the work- accomplished and for a thorough surveying of 
teachers to determine the grade level where each skill would be considered 
appropriate. Universes of objectives were sent out to each school and the 
process known as U.A.L. was begun. U.A.L. stood for Understanding, Accep- 
tance, and Level of Assessment. Each school was required to complete the 
U.A.L. process on a set of objectives sent to them. In most cases the 
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requirement was satisfied by one teaefcen, a small group of teachers, or the 
school principal completing the task. The U.A.L. profcess worked best when 
teachers did participate in the process. It was later discovered that* - 
several objectives that were "accepted" and "leveled" by principals were 
objectives which teachers singled out as unworkable and inappropriately 
leveled. Once again, a test instrument "got everyone's attention" and 
exposed what seemed Tike a harmless shortcut arising from the usual indif- 
ference. The U.A.L. process did serve to make Schools more^aware and more 
educated in the growing competency-based system as it also gave the SEP com- 
mittees more revision to do and the district some notion of what objectives'* 
to assess. 

The long-awaited SEP assessment! naturally followed. With the assis- 
tance of Dale Woolley, Difector of Pupil Personnel Services,. a schedule was 
developed so that a given SEP subject area would be tested district-wide 
every two years. A matrix sampling approach was devised with item sampling 
used on a 100 percent student sample. First assessments, included reading, 
computation, and math systems. Following assessments included language sys- 
tems and composition. Each year additional areas were tested. By 1977-78 
all SEP areas had been tested except for science, fine and practical arts, 
and doiiig. Teachers responded to these assessments in several ways. Those 
involved with the SEP or POE committees were enthusiastic and filled with 
anticipation. Many teachers at school sites were concerned that results not' 
be used to evaluate their effectiveness in the classroom. During this time 
of initial impact it was fortunate that due to matrix sampling tKe^district 
could not report results on individual students or classroom units, ^nly 
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grade^ level and school scores could be made^available. After several, years 
the SEP assessments had run their course and, due to budgetary constraints, 
were discontinued in 1979. 

•What took its place was' called Student Progress Monitoring or SPM. The 
new computer program written by data processing consultant Hal Roach, was- to 
permit the very individual student results that many teachers hoped for and 
that others feared. SEP assessments had been thought unnecessary by many 
teachers. They often objected to the expense of such testing and saw no 
"relevance" for the-"cl ass room teacher since they did not personally choose 
the objectives on which their students were tested nor did they receive 
classroom level results. The new SPM program was to solve these problems 
and quiet many of the objections. 

SPM began as a^pilot project at Paularino Elementary School where an 
interested, forward-thi nki ng- principal —'Bill Knight — asked for district 
funds to develop a small, individual student testing system with computer 
support. Based on his positive experience, a district-wide committee of 
teachers and principals agreed to meet with central office staff to describe 
the type of computer programs desired. This program proved to have the 
following advantages: 

(1) Individual teachers of groups of teachers were able to select 
specific objectives from the various SEP Universes, then develop 
tests using items from the district item banks and administer 
these tests- to their students.^ 

(2) The district would then machine-score the resulting answer sheets 
and provide reports back to schools indicating which students had 
mastered which objectives. 

/ 
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* (3) Teachers also had the option of writing their own objectives and 
items and assessing students with a totally "home-grown 11 test 
instrument. Although this option was not selected by a large 
number of teachers, it did give some teachers the feeling that 
they were not bound to use only the objectives that were in the 
district, banks. 

Although the SEP district assessment program was mandated during its 
duration, SPM started out as ?n optional testing program tnat individual 
teachers and groups of teachers were encouraged to use. Schools that were 
not performing well on the district SEP assessments were encouraged by cen- 
tral t £lrfice staff to use SPM to pinpoint thos^ students^who could benefit 
from added instruction on the particular outcomes in question. The intro- 
duction of SPM into the district was done on a low-key basis. Its use was 3 
voluntary arid increased each year after its introduction. When- the SEP dis- 
trict- assessment was suspended in 1&79, SPM, along with the .newly developed 
Minimum Graduation Proficiency Testing Program, became the main assessment 
tool of the district's competency-based education program. 

In 1976, California passed a piece of legislation called the Hart Bill 
which required the establishment of minimum graduation proficiencies in the 
basic skill subjects of reading, computation, and writing. The first class 
to be af fecte^ would be the class of 1981. Although many California dis- 
tricts were caught unaware and unprepared, Newport-Mesa was indeed ready for 
the challenge. Having complete "universes" of objectives in many subject 
areas including the basic skills, the district merely gathered community 
leaders, parents, administrators, SEP and POE chairpersons, teachers and 
some students to select important objectives for consideration as graduation 
proficiencies. Although such a process "worked"to a certain extent, a 
rather positive-thinking and ambitious group was turned- iQose on a task that 
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somewhat resembled the supermarket game show where contestants ayre given a 
market basket to legally "loot" all the merchandise they can in a specified 
period of time • It had all c the appeal and results of the proverbial excur- 
sion by the "kid in the candy stQre." After the first series of lengthy* 
meetings, the committee had identified over 1,200 objectives for -further ^ 
study as possible proficiencies! As would be'expected, this unwieldy list 
of skills was finally reduced to a more reasonable number while still being 
comprehensive of the curriculum areas to be tested. The final versions of 
the test were at last created by committees of secondary level department 
chairpersons both English and mathematics — who gulled much more limited 
lists of proficiencies from the original ones. Teachers would also select 
and write the items which would comprise the tests to be used on the 1981 
graduates. The tests would finally measure no more than 60 proficiencies. 

Having experienced conti nual * budget cutbacks due to severe declining 
enrollment, and penalizing state legislation as a result of Proposition 13, 
Newport -Mesa turned its limited budget and its CBE system to the proficien- 
cies and the support needed to guarantee student success. 7 The district's 
new Development Lab director, Nola* Roc.hel le, began development of several 
types of teacher support materials justified by student needs and demanded 
by teacher requests. Among the instructional materials created were: 

(1) Basic Skills Review Packets — individual review packets for each 
proficiency containing a pretest, review instruction and exer- 
cises, posttest, and (removable) answer key. All packets were 
directly linked to the item specifications handed out to teachers 
and distributed widely in the schools. 

(2) Proficiency Assistance Catalogue — a brochure listing both dis- 
trict-written and publisher materials found by teachers to be 
helpful in teaching to the tested skills. The Development Lab 
assists teachers in teh procurement and duplication of materials 
teachers request by phone or district mail. 
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In addition to assistance to the middle and high schools in the gradua- 
tion proficiency areas, the Development Lab began work with the elementary 
schools to articulate the proficiency skills at the lotfer grade levels — 
K-6. In the years 1980 and 1981, prof i ci'ency tests were devised for pre/ 
posttest use at all elementary grajle levels and^by 1982 were mandated by 
Dr. Loats, the Deputy Superintendent, as a non-optional testing program. 
Although some instructional materials are available to assist upper grade 
teachers, very little exists for primary. As it is affordable, more deve- 
lopment may occur in this area. 

The most impressive CBE support at the secondary level proficiencies 

i 

remains the creation of a summer school basic skills "lab." The six-week 
summer program was designed to take advantage of the pretest data available 
on students in grades 9-11 and to provide a wide variety of instructional, 
materials (including those mentioned" previously) and individual student 
attention for attending pupils. Staff selected to teach in the program 
attend important inservice sessions where the instructional materials are 
previewed and instructional approaches discussed. A testing room is set up 
on the summer school site where students take and retake mini -tests at the 
discretion of their lab instructors. Although enrichment is provided for 
students who wish to remain after they have mastered deficiencies, lab 
instructors have found that students prefer to "take care of business" and 
leave without the credits available for remaining the full six weeks. Seat 
time has been so deemphasized with this program that> students have little 
motivation when proficiency requirements have been satisfied. 
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Beyond the focus on the proficiencies in the basic skills and support- 
ing materials programs, the Board of Education continues to desire the ulti- 
mate fruition of the original CBE plans. This^year the district will admi- 
nister the first proficiency test that goes beyond the "minimum"— both in 
level of skills and as a district-imposed (rather than State-imposed) gradu- 
ation requirement. The class of 1988 will be responsible for social studies 
graduation proficiencies. Like basic skill areas, social studies .will be- 
come the beneficiary of not only our past proficiency experience but our 
seventeen-year goals. Through three superintendents and several turnovers 
in research and development personnel^, the blueprint and the vision have 
remained the same. 

Summary 

The CBE system of the Newport-Mesa Unified School District has been 1 
able to maintain momentum through several changes in school district manag- 
ing personnel. The Board of Education that exists in 1982 has only one mem- 
ber who was on the Board in 1965. Since unification, there have been three 
superintendents although the Deputy Superintendent (Dr. Loats) has remained 
the same during the entire period. Du? to the involvement of the staff from 
the inception of th CBE idea, there remains a widespread commitment to con- 
tinue the maintenance, revision, and new development needed to keep the CBE 
system viable. As a result of declining enrollments and budgets, recently- 
hired staff have been laid off and no new teacher contracts have been ten- 
dered since 1975. The majority of the remaining staff is committed to 
Newport-Mesa's CBE system. 
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The Newport-Mesa Unified School District has in place a competency- 
based education program anchored by adopted Board of Education policies. 
Although the major emphasis of the program at present is in the monitoring 
of progress made by students on a minimal level of desired proficiencies in 

the basic skills, efforts are beginning to^move the CBE system more into 

< 

higher levels of expectancy. The program's success has been indicated by 
several factors: 

(1) Teachers have become more involved in individualizing instruction 
in the basic skill areas. 

(2) The yearly performance .of students has also improved, both in 
terms of higher percentages of students demonstrating proficiency 
on the minimally required skills and 'also in termS of district 
scores on noYm-referenced tests, 

(3) Parents in the district have expressed greater satisfaction over 
communication from the district about the basic expectations for 
their children and how their children are doing. 

In the process; of implementing its competency-based education system, 
we have learned, sometimes the hard way, a number of important facts regard- 
ing the pioneering of such a program in the school district. The following 
is a brief summary of those^ facts: 

(1) It is possible to define the core of an instructional program^! 
teois of measurable objectives although one must be ready to 
devote a great amount of resources and energy to do this, 

(2) Although more resources, including time, will be expended, it is 
worth the time to- develop a system from within a schooKdistrict 
rather than to attempt to adopt a system of instructional outcomes 
taken from some other district. Staff is much more committed to 
the implementation of a system if they have had the opportunity to 
work on its development. "Home grown" products do have their me- 
rits. Even though the resources and motivation may not be avail- 
able, to a school district to spell out all of the curriculum in 
terms of measurable objectives, at least the basic core can be 
done. Such an understanding cannot be accomplished 9 in a brief 
period such as one or two years if staff is to be heavily 
involved. 
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(3) In the implementation of a competency-based education system, one 
cannot be too aware of staff concern for how the student assess- 
ment data will be used. The goals of such a system must be to 
assist students to learn. Once these goals are clear, it may be 
possible to use some of the information to develop evaluation 
agreements between teachers and principals. An exciting moment in 
Newport-Mesa !s history was when a teacher complained to district 
staff that she did not^see why she couldn't use test results from 
thq SPM program in her Stull Bill agreement (teacher evaluation 

" agreement). 

(4) In developincpneasurable objectives for any instructional program, 
you need to irigy£^nto the assessment phase before you can be 
assured of everyone's attentionspn the instructional objectives. 
Once assessment information has been collected, you must be pre- 
pared to- go back and reanalyze the content and scope of the vari- 
ous objectives that have been developed. 

(5) The task of scoring objective-based tests that require a "s-ub- 
scbre" for each objective is an enormous one. The use of a 
computer is almost mandatory. 

(6) When implementing a competency-based education system where mas- 
tery information is reported by individual objective rather than 
by total percent of correct items, it takes a very long time be- 
fore staff truly comprehends and becomes comfortable with this way 
of reporting. 



Questions We're Commonly fisked About Our System 

How much did it cost? Approximately $100,000 per year was devoted to 
developing Newport-Mesa's CBE system for the ten-year period, 1966-1976. 
From 1976 on, all development and operation costs of the system have been 
met through normal operating budgets of the various district offices, 
involved. 

Was the district's system developed primarily by outside consultants? 
No. Consultants were used to assist district staff in developing plans for 
the overall system, but actual work on the system was done by district 
staff. 
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Did you buy objectives and/or test items? No.. All objectives and 
items were developed by staff. Consultants were used on occasion to provide 

•or « i ' 

inservice training for staff relative to skills needed to develop good 
objectives .and items. 

Does your CBE system deal only with "minimum" expectations of the Board 
of Education, staff and parents ? No. Although the majority of required CBE 
tests in t,he district are oriented toward "quality control" or seeing to it 
that all students have mastered requi red* basal skills, some tests are used 
that measure skills that are not required of everyone but are "expected" of 
most students. 

Pitfalls/Mistakes to be Avoided 

In the rush to implement various testing programs, sometimes computer 
programs were developed in such haste that they did not follow standards 
normal ly fol 1 owed by the Data Center and were not always properly docu- 
mented. This fact made it more difficult to debug the programs and/or add 
to them later on. 

Although the school district emphasized staff involvement in the devel- 
opment of its CBE system, there were times when feedback was needed from 
schools that the principal was allowed to "vote" fo his/her staff rather 
than getting responses from each teacher. This always turned out to be a 
mistake since more often than not, it was those very same teachers who ob- 
ject e'd to decisions made from the surveys in which they had had no input. 
Factors Contributing to Success 

The continued use of district staff in the development of the system 
proved to be a major asset. Not only did this contribute to the final 
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acceptance of the system by the staff but it also provided the district with^ 
a cadre- of trained professionals to continue the development work. 
f Newport-Mesa's CBE system enjoyed, from the beginning, the support of 
the Board of Education and the Superintendent. The basics of the system are 
anchored in Board policy and hence will not, "go away" overnight. 

When turnover in the district. staff responsible for implementing and 
maintaining' the CBE system was necessary, replacements were selected who 
were trained^in criterion-referenced assessment and who were familiar with 
the district's philosophy .and policy in this area. 

Low turnover of principals and teachers in the district was an aid in 
that inservice training could be directed toward building on previously- 
trained skills -rather than on teaching; the basics. 
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CLARK COUNTY'S EXPERIENCE 

by The.ron Sv,ainston 

«*» 

District's Salient Features 

The Clark County School District was created in 1956 as part of a 
legislated reorganization to consolidate all of the school districts in the 
entire state of Nevada into seventeen districts — one for each county. 
Since its creation, the district grew from 20,000 students in 1956 to 
approximately 90,000 in 1982. The minority enrollment typically comprises 
25 percent of the total. In 1979 the State Legislature enacted an extensive 
property tax cut and placed stringent restrictions on public agency spending 
and revenues. In 1981 further legislation dramatically shifted the reverie 
base from real property taxes to retail sales taxes. The major nation*! 
recession which began in 1981 vividly demonstrated that this "tax shift" 
from real property to sales tax placed Nevada school finance at the mercy of 
economic ups and downs for the first time. In 1982 a $70 million school 
construction bond was defeated marking the first time the District had lost 
a bond election. 

Tax revolts, tax shifts, a defeated bond referendum and a major 
recession all served to strain the relationship between the community and 
the schools and confirm the value of an instructional management system that 
allows teachers, administrators, and board members to demonstrate 
accountability for acceptable student achievement. 
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Genesis of a Data-Based Instructional Improvement Movement 

In 1969 and 1970 a few central office administrators became increas- 
ingly uncomfortable with test scores and instructional inconsistencies. ^- 
Year after year the norm-referenced test results were below the mean and, in 
some grades, certain major subtests yielded percentile scores in the twen- 
ties and thirties. Initially, low.scores were rationalized by statements 
such as "Those tests don't measure what we teach," or, "The norms don't fit 
our district," or "You can't accurately measure school effectiveness where^ 
student transiency is so high." There were also concerns about gross incon- 
sistencies in curriculum and measured achievement from school to school . 

At that time, concern for student achievement was^kept from the fore- 
front through critical writers and legislation mandating minimum proficiency 
and demonstrated accountability in neighboring states. Also, at that time, 
a district administratire'reorg^nization. aimed at centralization placed the 
total supervision of all schools under two assistant superintendents— one 
for elementary and one for secondary. Thus with the need demonstrated and 
the organizational structu-e put in place, the stage was set for a major 
thrust toward data-based instructional improvement. 

Shape and Direction of the Movement 

The Assistant Superintendent for Elementary. Education, encouraged and 
assisted by certain other administrators, assumed the leadership for the 
design and implementation of a comprehensive system for instructional 
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management and improvement beginning in 1970. It was determined that in 
light of the rapidly changing demographic composition of the district, any 
significant and lasting improvement would have to be centrally developed and 
directed. While the system's design and programs were not mandated by the 
superintendent or the board of school trustees, so long as too many feathers 
were not ruffled or too many financial demands made, they were usually sup- 
portive. At one point, as some principals began to feel uncomfortable with 
the demands for' change and accountability, as they visited with the super- 
intendent they asked him for relief. Initially he sympathized with them, 
but as he was made more fully^aware of the system's potential he became a 
staunch supporter. 

After approximately seven years of development and utilization in the 
district's elementary schools, in 1978 a new superintendent directed that 
the system be adapted for and implemented to a limited degree in the secon- 
dary schools. This required considerably less development time because a 
model was in.-place and operating. However, that superintendent resigned in 
1981 and the extent of Secondary development and implementation became 
uncertain. 

From the beginning, certain assumptions about curriculum, evaluations- 
staff development, management and school administration, and the role of 
educational research provided continuity of approach and design. As the 

movement extended in years and people, these assumptions became increasingly 

\ 

important. Perhaps the three most basic and far-reaching assumptions were: 

j* - - - .. .. 

1. Goals and objectives need to be clearly written and widely 
communicated. 
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2. Means must be provided and used to assess the degree to which 
objectives are attained, 

3. All assessment should culminate in program improvement decisions. 

It was always assumed that each component of the overall school and 

instructional management system would be designed to fit into and facilitate 

one or more of the above three basic assumptions. Those three came to be 

expressed commonly as "What should be, 11 "What is," and "How to reduce*the 

difference." It was also assumed that the probability of objectives being 

attained would be greatly increased if each individual were held accountable 

through a modified management by objectives and results approach. 

■ 

Some of the other assumptions that were made consistently were as 
f ol 1 ows : x 

1. Curriculum guides should be developed in the district and should 
define the specifics of curriculum content through learner behavior 
objectives for each subject. 

2. Effective instruction does have research verified characteristics, 
and instructional programs and teacher performance can be evaluated 
objectively. 

3. Teacher effectiveness can be improved through skilled supervision 
and relevant -staff development. 

4. The principal can play a key role as an instructional leader within 
a school and will effectively fill this role if given the skills, 
motivation and direction to do so. 

Instructional Management System Design 

The system has three major components or subsystems. The first com- 
ponent consists of goals, objectives, and evaluative criteria ("What should 
be"). These range in scope and specificity from a simple math skill in the 
Kindergarten Curriculum Guide to annual priority goals adopted by the 
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superintendent and the board of school trustees. At the school level the 
heartof this component is a set of objectives and standards for school 
operation known as Elements of Quality, f~ 
The' ten basic elements listed below have become widely accepted and are 
commonly used as criteria in planning for and ev- -uting instruction. Each 
of the ten elements (an eleventh was added for student activities in secon-' 
dary schools) has sub-objectives or specific evaluative criteria. These, 
specific criteria serve as guides for curricular and instructional planning 
and both self and program evaluation. The first five also serve as a set of 
observable criteria to guide principals in eveluating teacher, performance* 
All of the elements are used By principals in making an annual assessment of 
their school (note Element 6) and by the supervisors of principals in making 
an annual evaluation of each principal's performance. 

Clark County School District 
ELEMENTS OF QUALITY FOR ELEMENTARY SCHOOLS- 

Note: for brevity, the subobjecti ves and evaluative criteria have either 
been omitted or printed in abstracted form. 

1. Curriculum guides serve as the basis for'dassroom instruction. 
Lesson objectives and learning activities are planned in accordance 
with specified skills and concepts contained in Clark County School 
District curriculum guides. Content of textbooks, etc., is used 
selectively to teach and reinforce skills and concepts specified in 
curriculum guides. 

2. Student achievement is commensurate with ability or other established 
expectancies. 

3 # Individual differences in the educational needs of students are 
identified and- appropriately met. 
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The learning needs of students are assessed in relation to esta- 
blished objectives. . Students are grouped for instruction according 
to assessed needs* Instruction is adjusted to student learning- 
rates. Opportunities are provided for students to use their most 
effective ways of learning. Learning progress is monitored- and 
recorded. - 

Provision is made for the social and emotional development of stu- 
dents. Positive direction and reinforcement are the primary means 
for motivating students. Students respond positively to teacher model 
and direction. Students remain attentive to their work. Students 
display positive behavior when interacting with each other. Students 
are provided opportunities to initiate* direct, and evaluate 'some of 
their own learning activities. Student talk and movement are appro- 
priate for the learning activity. 

Instructional methods are consistent with established objectives and 
proven principles oT learning. * 

The teacher sets a positive climate for learning (stimulates student 
excitement, anticipation, curiosity, etc.). The teacher clearly com- 
municates lesson objectives and their importance to students. The . 
teaching activities are appropriate for concept, skill or positive 
behavior lesson objectives. The teacher clearly communicates direc- 
tions for follow-up activities. Teacher questioning strategies 
(questipns, responses, reaction) facilitate the development of think- 
ing skills (literal, interpretative and critical)/ Resources are 
appropriately used for achieving lesson objectives. The physical 
environment of the classroom is well organized and designed by the 
teacher to enhance, stimulate and extend learning. 

A management system providing for needs assessrrant, priority objec- 
tives and plans, monitoring and evaluation by results is effectively 
used by the principal and teachers. 

Principals are involved and involve their staff members individually 
in a structured assessment, priority setting, planning, evaluating 
and reporting process' for improving performance results in relation 
to established criteria. 

Personnel management procedures prescribed by law, regulation, and 
contract are effectively administered by the principal. 

The principal makes (and records as appropriate) frequent visits to 
classrooms to directly observe the instructional program and holds 
individual conferences with teachers as needed to discuss their 
priority objectives and plans, their progress toward objectives, 
needed change and assistance and overall performance results. 

Staff effectiveness is promoted by the principal through proper 
application of proven principles of leadership and management. 
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The principal provides for staff involvement, promotes an open cli- 
mate, provides training and assistance as needed, reinforces good 
perforfftance of. the staff, and accurately assesses and appropriately 
responds to measured faculty opinion regarding the operation of the 
school . 

9. Community confidence in the school is established and maintained* 
Parents are kept well informed regarding the school, its objectives, 
programs and procedures, are provided convenient means to express 
their opinion anci suggestions regardi ng Jthe school, are kept well 
informed regarding their child's school program and progress, and are 
provided. means to be involved at the school and supportive of its 
objectives and programs, 

10. Management organization and procedures for the school are clearly 
written, effective and consistent with the established procedures and 
regulations of the District, 

Necessary and standard school regulations and management procedures 
are clearly written in a staff handbook. Provision is made by the 
principal to monitor and evaluate the management functions of the 
school to identify exceptions to established standards and procedures 
and to appropriately deal with management exceptions. 



A second .component consists of both formative and summative measures of 
the extent that objectives are achieved ("What is"). Norm-referenced 
(nationally standardized) tests are administered in grades three, six, eight 
and eleven. Locally developed criterion referenced tests are used in read- 
ing and mattr in three mo'des. One CRT form is used a£ a general placement 
test generally v at the beginning of the school year. Another CRT form, is a 
series' of specific -diagnostic instruments used to identify specific learning 
needs for individual students. The third CRT form is an end-of-year measure 
of mastery of specific instructionalvobjectives. Other measures of "What 

A 

is" include structured surveys of parents, student, teacher and principal 
opinions and recorded direct observations of instructional personnel and 
student behavior utilizing specific observable criteria growing out of the 
Elements of Quality. The opinion surveys are machine-scored and by tomputer 
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processing school personnel are provided data through printouts keyed to the 
Elements of Quality. 

The third major component of the system directly addresses- the chal- 
lenge of data-based instructional improvement ("Reducing the difference" 
between what should be and what is). This component includes a framework 
for goal setting, priority planning, monitoring and staff development. 
Utilizing the data base provided by comparing ."What is" with "What should 
be", each level of tfie district organization from teacher to superintendent 
and school board selects as priorities for change a limited number of objec- 
tives which are/realistically attainable and offer' the greatest probability 
for instructional improvement. Each of these priorities is written in a 
universal format to promote simplicity and common understanding. The format 
includes statements of identified need, objectives, activities to achieve 
the objectives, the means for evaluation, the person(s) responsible and 
finally a statement of evaluation added at the end of the annual cycle. To 
encourage the selection of significant priorities, a direct relationship 
between achieving personal priority objectives and personnel performance 
evaluation is carefully avoided. 

ResOlts 

Standardized test results have clearly shown that the district's 
efforts to improve. measured student achievement were effective. Elementary 
achievement at both the primary and intermediate grade levels increased 
approximately 20 percentile points over a period of. seven to ten years from 
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1970. For example, the sixth grade total reading and math percentile scores 

had increased from 42 and 38 in 1970-71, to 60 and 64 in 1981-82. The 

second grade reading and math percentile scores had increased from 50 in 

1970-71 to 71, and 76 by 1976-7^. In 1978 standardized testing was moved 

from second to third grade where in 1981-82 th^ 7 total reading percentile was 

64 and math was 71. 

Since the secondary schools did not begin to utilize a comparable 

instructional management system until later, direct comparisons between 

elementary and secondary achievement are not so meaningful. However, it is 

reasonable to assume that elementary students would carry some of their. 

higher achievement into the secondary grades. The eighth grade reeling and 

math percentile scores increased from 40 and 36 in 1975-76, to 59 and 65 in 

i 

1981-82 and- the eleventh grade i-ncreased to 53 and 57 by 1981-82. 

The results of district-developed criterion referenced measures of ele- 
mentary student proficiency have been used to group students within a class 
forr instruction, to diagnose individual student learning needs and to eval- 
uate instructional programs in terms of student mastery of instructional 
objectives. Care has been taken to avoid using the criterion-referenced v 
placement and diagnostic tests as measures of' student performance for 
accountability purposes. By machine-scoring placement and end-of-year cri- 
terion referenced tests we can give printouts to individual teachers and 
principals. These printouts provide extensive analysis data by student, 
classroom (teacher), grade, school, and district. m J t 

An analysis of the responses to el ementary teacher and parent opinion 
surveys which are criterion referenced to the Elements of Quality reveals 
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certain attitude patterns that have been consistently positive for several 
years. Teachers and parents respond to each item in the survey instrument 
on a five-point Likert-type scale. The weighted point scores could range 
from a low of 1.00 (strongly disagree) to a high of 5.00 (strongly agree). 
From 1978-79 to 1981-82 the mean weighted response on the teacher instrument 
increased from 3.36 to 3.46 for Elements 6 through 10. The' mean weighted 
response on parent survey remained very stable, but high with a mean of 4.36 
in' 1978-79 and 4.37 in 1981-82. .Certain items on each survey have special 
significance. For example, on the parent survey, "The teachers use effec- 
tive methods and materials to help my child learn" yielded 4.46 in 1978-79 
and 4.47 in 1981-82 with an average 14,000 parents responding, representing 
all of the 71 elementary schools. Two of the most significant items'on the 
teacher survey were "My teaching performance is accurately assessed relative 
to District-estabished criteria" (3.51 in 1978-79 and 3.55'in 1981-82) and 
"The principal's supervision of my classroom performance results in improved 
instruction" (3.29 in 1978-79 and -3.44 in 1981-82). An average of 1,950 
teachers responded which was nearly 100 percent of the total number of * 
elementary teachers on staff. One of the challenges of a highly structured 
and centralized instructional management and accountability system such as 
ours is managing the affective responseof teachers. The teacher opinion 
survey is used annually -to allow individual principals and district admini- 
strators to monitor' the attitudes and feelings of teachers. Both the 
teacher and parent surveys are administered in a manner to encouj'age^oBjec^ 
tivity through anonymity of respondents. Since each item in <the survey in- 
struments (35 for teachers and 15 for parents) is a clear statement of a 
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desirable condition or objective to be achieved, they represent a directing 
force as well as a means to measure attitudes* 

A Unique Feature ' 

y 

A very significant and somewhat unique characteristic of /the Clark 
County experience deserves special consideration. Many top level Clark 
County administrators believe that it is the single most vital difference 
between what might have been a passing experiment of limited impact and one 
that has endured and has had a profound influence in dramatically reshaping 
the instructional program of a very large school district. The management 
of all instructional improvement efforts was deliberately linked with the 
overall direct management of schools . The following examples will serve to 
illustrate this point. 

When one independent department was responsible for developing curri- 
culum and others were responsible for implementation, the use of curriculum 
guides was considered to be optional and many principals and teachers did 
not take them seriously. However, when key Tine administrators assumed 
responsibility for the development, implementation and success of the curri- 
culum, a new standard was set. One of the criteria used to evaluate the 
performance of principals and teachers, was the extent that curriculum guides 
were used as a basis for instruction. (See Element of Quality No. 1.) A 
uniform curriculum became a reality. 
x Prior to the early 1970's, test scores were not taken seriously except 
by a,few people and mostly by the central office staff responsible for 
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testing arid evaluation. However, when the comprehensive school and instruc- 
tional management system was developed, acceptable test scores became one of 
the criteria not only for evaluating instructional programs, but also for 
evaluating the performance of instructional personnel. (See Element of 
Quality No, 2.) Here an important distinction was attempted. An adage was 
coined that "When tests are used as weapons of supervision rather than tools 
of instructional improvement, they soon become useless for either purpose." 
Rather than an individual's effectiveness being evaluated upon reaching a 
norm or an absolute standard, they were evaluated upon showing needed or. 
significant improvement. This permitted teachers and principals in tradi- 
tionally low achieving areas to be winners and" it required those who hap- 
pened to be in traditionally high achieving areas to do more than exceed the 
district means. It was reasoned that if the bottom line of the school busi- 
^ ^L ess 1S student achievement, those who control the day-to-day operation of 
the business should be directly accountable for the bottom line . 

It was determined that principals and their direct supervisors would be 
the first line instructional leaders. They would plan and conduct or direct 
staff jjfcvelopment for themselves and teachers. They would be involved in 
curriculum development and all curriculum implementation and instructional 
improvement programs would flow through theme 



In a continuing sequence of devel opment/ implementation and refinement 
since 1969, the aspect of the system whicn currently offers the greatest 



Challenges and Recommendations 
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challenge and promise is staff development. For example, considering the 
classroom teacher and school principal we can now describe quite well the 
characteristics of good performance ("What should be") and with considerable 
accuracy assess the performance of a given individual skill and motivation. 
An additional challenge of a highly structured system is providing for 
upgrading and renewal of all the components to ensure that it is dynamic and 
rel event. 

Certain recommendations seem to be appropriate for others who might ^/ 
consider utilizing any part of the Clark County system or experience. It is 
important to assess the philosophy or assumptions of key leaders to ensure 
that there is compatabil i ty with goals and approaches that are developed, ^ 

Objectives are most basic ancl should be clearly written for all levels of 

~* 

the organization and generally accepted by those responsible for attain- 
ment. There are clear time advantages of adapting existing .models where 
they can be found rather than starting with nothing. Formal endorsement and 
support from the Superintendent and the Board of Trustees is ultimately ne- 
cessary and should be sought early. Heavy involvement in design and devel- 
opment by those staff most directly affected helps to ensure acceptance and 
commitment. 

Questions Commonly Asked About the System 

Administrators who work with the Clark County system are most often 
asked if the management and accountability system has made a positive dif- 
ference in teacher and principal behavior and student achievement. Princi- 
pals and teachers are much more structured and objective in planning and 
evaluating instruction as a direct consequence of the system. While it is 
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not always possible to quantify the exact degree, it is possible to demon- 
strate that the system is the most significant cause for dramatic improve- 
ment in test scores, our principals and teachers are now far more more 
sensitive to the need to ask questions of themselves such as "What changes 
can we make which will result in the greatest possible improvement in 
program quality, and how will we demonstrate or measure the desired improve- 
ment?" This represents not only a more objective attitude and approach pat- 
tern, but has led to a dramatic increase in planning and evaluation skills 
and knowledge at the school level. 

The question of whether the increased centralized structure has reduced 
individuality is often asked. Great care has been taken to place emphasis 
upon specifying curricular objectives and measuring results rather than lim- 
iting methods and individual techniques. For example, the distritt care- 
fully avoids single textbook adoptions and allows teucher* and principals in 

T 

individual schools to select texts from an .approved list of two to five 
books for each subject. Even wirier .1 ndividual discretion is allowed in the 
selection of supplementary instructional materials. 

The question of how widely the system is accepted is also raised.- The 
fact that the system, has spread from elenantary to' secondary schools and has 
endured at the elementary level for more than ten ^ears says something about 
its acceptability. However, the system, has been and continues to be re- 
sisted by certain people. The line of people demanding or volunteer'' i.-, to 
be held more accountable is never long. Approaches as such an. ual opinion 
surveys and widespread involvement in the development of system components 
have increased the acceptability of the system. The system survived a 
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lawsuit by an employee group which originally saw it as a threat to 
employment security. 
Pitfalls/Mistakes to be Avoided 

Perhaps the most serious pitfall to be avoided is the tendency to use 
such a management system almost entirely as a tool (or worse still, a 
weapon) for controlling at the expense of planning, organizing and motivat- 
ing functions. To avoid this, great care must be taken to ensure that those 
who administer the system understand and apply the principles of management 
and leadership upon which the system is based and without which the system 
will die *of its own weight. 

Another pitfall to be avoided is moving too far too'soon. Components 
of the system should be structured in detail and field-tested before wide- 
spread implementation is attempted. A master plan for the entire system is 
highly desirable in order for each component to be seen in proper perspec- 
tive. For example, a criterion-referenced test far sixth grade mathematics 
should not be viewed as boards and nails, but as part of a building that caR 
be viewed in blueprint elevations and floor plans. To move too far too soon 
sets up resistance Shockwaves stemming from misunderstanding and anxiety 
about the value of such a system to either the employee or the students. 

Special attention should be given to keeping paperwork to a minimum and 
the major focus on what happens in the classroom compared with what should 
happen. 

Factors Contributing to Success 

I believe that a major factor contributing to the success of the Clark 
County system is consistency of design and application. Because the system 
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rests upon a clearly defined philosophy and research-proven principles, it 
has weathered well the shifting winds and storms of criticism and passing 
"latest" innovations- For example, when "Theory Z" and "Quality Circles 11 
came upon the scene, the system had a place for them. When the Beginning 
Teacher Evaluation Study became a major topic of discussion, the -system 
easily accomodated the majof findings and implications. 

The overall system design rests upon a core of overriding criteria de- 
fining quality at the school level, and these Elements of Quality are mani- 
fest throughout the entire system. We measure almost everything is measured 
in light of its consistency with these criteria. The application and opera- 
tion of the system rests upon certain proven principles of management, and 
administrative behavior- is evaluated in light of consistency with these 
principles (process) and the criteria defining quality education (product), 
because this consistency has been preserved over an extended period of time, 

the system and its components have become second nature to those who have 

t 

"grown up" with our system. 
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VALLEJO'S EXPERIENCE: LINKING ' - 
TESTING WITH INSTRUCTIONAL IMPROVEMENT 

by Joan McDonald 

Introduction 

The Vallejo City Unified School District is. a medium-sized California 
school district (14,000-15,000 students) located in an urban, multi-ethnic 
community. Over 50 percent of the students are members of a minority 
group. Black students represent the largest minority population, followed 
by a fast growing Filipino population and an increasing Hispanic popula- 
tion. 

Vallejo is a low-income community. Mare Island Naval Shipyard is the 
major employer in the city; the school district is the second major em- 
ployer. Many of the students' parents are unemployed or have low-paying 
jobs. Most students enter school without preschool experience. 

Vallejo's students have many needs, both academically and economi- 
cally. The primary goal of the district is to provide the students with an 
excellent education and to see that they reach their fullest potential. The 
same goal probably exists in all school districts. This is the story of one 
district's efforts to make the vision a reality through instructional im- 
provement. The use of test data became a catalyst; for analyzing instruc- 
tional problems. 

The vision that the superintendent and his staff have is a district 
with a clearly expressed philosophy and direction. Too often a school dis- 
trict does not take the time to articulate a philosophy and to establish and 
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communicate clear directions both within the district and to the public. It 
is assumed that everybody wants the "best for kids" and knows how to bring 
that about. Part of Vallejo's vision has been to establish and articulate a 
clear sense of purpose. 

Improved instruction was a part of the vision. The district's defini- 
tion of improved instruction included: 1) a district curriculum where ob- 
jectives, learning activities and evaluation procedures match and are con- 
sistent; 2) schools in which principals and teachers engaged in ongoing ef- 
forts to analyze and improve teaching; and 3) the use of test results to 
improve instruction. 

A long-range plan for curriculum development was initiated. A district 
professional development center was established to research, teach and pro- 
mote sound instructional practices. Ongoing efforts were made to become 
familiar with the purposes, content and uses of tests and test results. 
This article will describe in chronological order the problems that preci- 
pitated the actions taken, a description of some of the solutions and a sum- 
mary of lessons learned. ■ 

The Beginning 

The California Assessment Program (CAP) was a major influence propel- 
ling the Vallejo School District to use evaluation information to improv^ 

j 

instruction. CAP was initiated in 1974, toward the end of an era of decen- 
tralization in the Vallejo School District. Between 1965-1975, school sites 
had considerable autonomy and had assumed responsibility for curriculum 
development and staff development. Schools had experimented with a variety 
of organizational schemes such as team teaching, departmentalization and 
non-graded classrooms and had tried various schedules for students. During 
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this decade, decentralization was actively encouraged to promote school -site 
autonomy and decision making.' District-wide coordination took a back seat 
to experimentation and the development of school-site ownership of instruc- 
tional programs, 

r ') 

When CAP came on the scene, it assumed center stage. Test scores made 

i 

front page headlines and became "the agenda" for disgruntled members of the 
governing board and the community who were already critical of the schools, 
CAP was a catalyst. The educational community had no choice but to deal 
with CAP, though initially it would have preferred not to. 
Problems That Arose 

Dealing with CAP testing and its results productively was difficult for 
us for several reasons.- First, the previous decentralization of our school" 
system impeded any coordinated and comprehensive effort to use the results. 
Second, CAP testing became a highly emotional issue both with the public and 
within the school district. Third, .our school staffs had limited knowledge 
about how to use test results to improve the instructional program. 

The previous decent ral izati on of the school district contributed to our 
problems because there were no consistent district -wide instructional prac- 
tices and clear curriculum outlines. Additionally, schools had become ac- 
customed to making decisions for themselves without necessarily coordinating 
their efforts with other schools. No district -wide statement of the curri- 
culum existed to tell teachers what skills should be addressed when and in 
what formats . Title I project elementary schools had done more in terms of 
defining the curriculum for themselves, but they had vtorked in isolation 
from one another. So, in fact, each Title I school used a different skill 
continuum. Non-project schools typically had no curriculum frameworks. 
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Textbooks provided the curriculum, and teachers taught what the teacher's 

... T 

guide prescribed and used the methods described there. A further problem 
♦ 

was that schools made autonomous decisions about textbook selection and, in 
fact, many different series were being used throughout the district. 

The emotionalism surrounding CAP interfered with its being. used as a 

tool for improving instruction. Some of the most severe critics of schools 

or 

seized on CAP as a vehicle for casting criticism and blame on the schools. 
The educational community responded defensively to past and current misuses 
of test information. Ignorance increased emotionalism on both sides. 

School staffs had limited knowledge about the purpose-, content and 
usefulness of the CAP tests. The district leadership had not emphasized the 
use! of test results. Since the district curriculum was not defined, prin- 
cipals did not have identified expectations to help them use test results to 
improve the school's instructional program. Neither principals nor teachers 
had training to equip them to properly analyze, interpret and use test re- 
sults for program planning. 

Central Office Decisi ons About the CAP Test: School-site Analysis 

1 ~ , 

The central office leaders decided that a district -wide effort to use 

test information to improve instruction had to be initiated. The plan for 
accomplishing the goal included developing awareness on the part of princi- 
. pals, training principals in the use of test results, and providing direc- 
tion for school-site analysis and planning. This process led to a series of 
long-range efforts in the area of curriculum and instruction. 
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A centralized plan for using test information vras developed to initiate 
some consistent processes across the district for dealing with test re- 
sults. We emphasized that those in central office leadership roles believed 
that test data was important to program development and evaluation. As will 
be described in the following sections, our plan moved methodically from 

V 

promoting awareness, to training, to requiring application of knowledge at 
the school site. Use of test data was the intended outcome, but those who 
were providing leadership were aware that some emotional and awareness 
issues had to "be addressed first. Although site managers did not respond 
Pnt hnt:i AQt-ir^l ly in thp hfrg inning , a growing app rpri a tinn for the value at — 
using test data did develop.- In the central office, we initiated a process 
to ensure that school -site staffs used test data to improve the instruc- 
tional program. 

The first step in developing principal awareness about the power and 
uses* of test results was to overcome the emotionalism that surrounded the 
issue. The most immediate reaction of principals was highly defensive and 
resistant. Without even reviewing the test/results of their schools, a 
majority : of the principals rejected them. Principals claimed loudly, "CAP 
didn't test what we teach, " CAP is "biased" and "irrelevant." Another favo- 
rite cry was, "CAP was created by test makers wholiave little or no know- 
ledge about schools and what we do." The fact that .the district had a low 

_ — .. ' 
percentile ranking even in relationship to other districts in its comparison 

score band didn't help a bit. The fact that the local newspapers headlined 
stories that governing board members were accusing the schools of failure 
only served to further hamper communication. 
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The district leadership realized that further emotional discussion 
would only exaggerate the problem. We made plans to provide training to 
"principals, selected teachers and the governing board so they could begin 
dealing with CAP productively. Inservice topics focused on information 
about the test itself, skills to analyze test data and strategies for inter- 
preting and using results to plan for Instructional improvement. 

The central office administrators contacted the Office of Program 
Evaluation in the State Department of Education. Consultants from the State 
office came* to Vallejo, met with the district staff and principals and con- 
duct ed-a^et-ies^f^wareftes^— ievel-sessions^n^e-pur^ose-^d_develfipinenL — 
of CAP. Test score reading and interpretation were also essential compo- 
nents of the awareness level training provided by the state staff. The 
information contained in the workshops was essential in terms of providing a 
knowledge base to begin dealing with CAP results', yet the training itself 
did little to affect the defensive attitudes of principals and teachers. 

The next activity we initiated had the most powerful impact on school 
staffs. District j)ffJce>^ministrators drew up a three-step process in 
which school staff were required to work through and submit in writing an 
analysis of their test data and a plan of action. The process involved 
becoming familiar with the content of the CAP test, then analyzing trie 

4 

school's program and finally identifying 1 di recti ons for improvement. The 
directions were then reported to the governing board. 

This school -site analysis process required school staffs to use the 
Test Content Specifications provided by the State Office of Evaluation tc 
become familiar with the actual skills tested by CAP. The purpose of this 
component of the process was to have/teachers and principals answer the 
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question, "Are we teaching what is being tested? 11 This was an important 

i 

step because it caused principals and teachers, for the first time, to deal 
specifically and objectively (rather than emotionally) with what CAP tested 
in relation to their school 's instructional program. The argument that 
"this doesn't test what we teach" faded abruptly. 
ONE SCHOOL'S STORY 

The following is one principal's description of how the school- 
site analysis process worked at his school. The school described here 
was actually located on two campuses and served by one principal. The 
student~p;opuTat1on~was madfcu^ 

a high percentage of students who demonstrated low academic achievement 

on standardized tests. At that timei the school also had the highest 

percent of AFDC (Aid to Families with Dependent Children) students. 

The Principal's Description 

Most of the school -site analysis activities were related t9 the 
use of CAP results. Other standardized tests were also administered to 
evaluate our Title I program, but the use ^of other test data at this 
point in our history was primarily a result of transferring knowledge 
gained by using CAP results to other testing programs, 

The Instructional Associdce- (a non-classroom teacher paid for out 
of Title I funds), selected teachers and I reviewed the test results to 
identify areas of concern. Areas of concern were defined as those 
areas that either seemed to be especially discrepant with overall test 
results or those areas where their results seeded discrepant given the 
amount of instruction that had been provided a^d the results expected. 
It was also decided that no more than three sreas would be selected in 
reading and three in math for a more In -depth analysis. 

The Instructional Associate and I used the State manuals and other 
materials we received during the training provided by the State 
consultants to: 

- identify and define the content being tested in the areas 
that had been identified; 
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- restate the learnings that were being tested in our own words 
and in terminology familiar to our staff; 

- prepare a task analysis of each of the skill areas. 

1 

The above information was shared with all staff members at regu- 
larly scheduled meetings. A decision was made to familiarize all of 
the teachers with CAP, not just the teachers of students in the grade 
levels that were tested. We wanted the entire staff to share the re- 
sponsibility for improving our instructional program. 

After the skill areas tested by CAP were carefully analyzed and 
shared with the staff, they were matched with the appropriate objec- 
tives and evaluation instruments in the reading and math curriculum 
management systems being used at our school. Some of the teachers were 
involved in validating. the match between the skills areas being tested 
by CAP and the skills being taught through the use of the reading and 
math textbooks at the school . 

All teachers worked individually or in small groups to develop 
prototype lessons and activities for teaching the identified skills at 
different levels of difficulty. These products were shared among the 
teachers for use in classroom instruction. The lessons and activities 
were not the only results of this part of the process; it also produced 
a renewed sense of purpose and teamwork among staff members. 

In addition to improving their lessons for students, teachers also 
began working on teaching their students test-taking techniques. 
Teachers examined the testing formats used by CAP (multiple-choice, 
fill-in-the-blanks, etc.) and incorporated those formats into the prac- 
tice activities they built into their lessons. Teachers also familiar- 
ized themselves with some of the key words used in test directions and 
taught them to their students. The Instructional Associate and I de- 
veloped practice activities and conducted practice sessions for groups 
of ten to fifteen students. 

Outcomes at the School 

A very important outcome of the school-site analysis process at 
the school described here was the change in attitude toward testing. 
Staff meetings were used to discuss the value .of using a testing pro- 
gram to plan instructional improvement/ Teachers came to the conclu- 
sion that testing is an important component of a school ^ogram, that 
analyzing test results could help them monitor and modify instruction 
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and that test taking itself is a valuable skill for students to 
acquire. 

Test scores at the school described did begin to improve as a 
result of the school-site analysis process. Miracles did not occur. 
Test scores were not immediately and dramatically much higher. The 
instructional program did improve, the school staff developed a more 
positive attitude toward testing and a more cohesive plan for using 
test results, and test scores reflected those efforts. The principal's 
active leadership, interest and involvement provided a powerful message 
to the staff and led to their active involvement in using test data to 
improve instruction. 

District Outcomes Leading to New Directions : 
Curriculum, Staff Development and CRT's 

There were several outcomes that resulted from developing a greater 

awareness about CAP, providing training to help principals and teachers use 

/ 

/ 

test results for instructional improvement and requiring each school to use 
a school-site analysis process. 

The first outcome was an improved attitude on the part of the educa- 
tional community with regard to the potential uses of test information. The 
educational community as a whole became more objective, analytical and con- 
fident as it developed an understanding of information and the skills to act 
on this information. 

A second outcome was a growing awareness and concern for continuity, 
consistency and coordination of the curriculum that surfaced both within 
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schools and across schools. School staffs also became aware of specific 
skill areas that needed to be given more attention because students were 
demonstrating difficulty with them on tests. 

The outcomes described above led to some new directions for the dis- 
trict. The realization that there was no well-defined district curriculi 
resulted in a substantial curriculum development effort. 

Within three years district continua in reading/math/language were 
developed. The district also established- a Professional Development Center 
to provide extensive training to principals and teachers- in improving their 
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instructional skills. Attention was given to improving testing procedures 
throughout the district. Two years later criterion-referenced tests and 
student profile cards were introduced to complete the curriculum management 
system in those skill areas. 
. Curriculum Development 

The closer look at the instructional program that followed, a greater 
emphasis on using test results made everyone very aware that no district- 
wide statement of what skills were to be taught to whom and when existed. . 
With the advent of the "Hart Bill" proficiency test movement, the district 
moved to develop the necessary district-wide statement of skills. Skills-- 
continua were written in reading, language and math, including the identi- 
fication of proficiency skills developed throughout the K-12 curriculum in 
basic skills. 

For the first time, teachers came together across grade levels and ham- 
mered out the cjontinua. Certainly, those who were most centrally involved 
in that effort ^earned the greatest amount from the experience. However, as 

\ 
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those continua were introduced to staffs and field tested, a wider and wider 
segment of the teachers was affected. The first steps toward improving 
instruction had been taken, as throughout the district, individual teachers 
were becoming more specific and articulate about the skills being taught. 
The greatest benefit was a beginning recognition of a move toward a dis- 
trict-wide statement of r ; *riculum. 1 

The next step was the development of the Criterion Referenced Jests 
(CRT's) in relation to designated essential skills in the continua. The 
requirement that teachers test their children and record their progress on a 
district-wide (K-6) student profile card finally moved the continua into 
prominence as the basis for instruction. The use of the CRT's and requiring 
teachers to record skill mastery on student profile cards can be identified 
as crucial to the institutionalizing of the continua of skills. Prior to 
that time, only some of the teachers truly felt bound to teach the district 
curriculum. 

Inservice training for teachers was provided on school sites by central 
office personnel assisted by teachers who had been involved in the develop- 
ment and subsequent revision of the continua and the CRT's. The more in- 
volved a staff became, the more aware they became of the need to review 
together the task analysis and instructional strategies involved in teaching 
the essential skills. Now, staffs are identifying their instructional areas :> 
of need through CRT's as well as CAP and CTBS. In addition, they are also 
identifying areas of need for staff development. Instructional improvement 
is, beginning to show results in test scores. 
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Principals became aware of the complexity of truly bringing about 
change as they provided' leadership to their staffs in implementing the dis- 
trict curriculum management system. One principal tells the following story 
which ^illustrates some of the challenges that confronted her and the solu- 
tions she' tried at her 4-6 school. 

ONE SCHOOL'S STORY 1 

The Principal's Description 

In the spring of 1981, the district curriculum management system 
package arrived complete with continua, CRT's and student profile 
cards. Our staff spent the spring and summer playing with a variety of 
ways to introduce and teach its use to our staff. At this point, we 
were primarily dealing with the mechanics of using it. How would 
teachers get CRT's? How could teachers manage 90 profile cards? How 
could we insure that teaching would go on prior to testing? How could 
we streamline and facilitate the duplication and storing f>f CRT's? 

We finally developed what we considered to be an excellent, almost 
foolproof system for management, and an enlightened inservice program 
with audiovisuals, songs and dances, etc., on how this foolproof system 
would work was developed and presented in the fall. The inservice was 
heralded as a great success. Teachers seemed delighted with the organ- 
ization and toted their boxes of 90 cards (one for reading, math and 
language for each student) to their rooms filled with enthusiasm and 
good will. 

The first hint of any disaffection was evident in November of 1981 
when teachers were required to turn in their student profile cards for 
review by the principal for the first time. Comments like, "If I could 
use my time teaching instea5 of marking cards, my kids would learn 
something " were common. My review of the cards showed haphazard mark- 
ing, no uniformity of skills addressed, and general lack of care taken 
in their marking. This response came from a generally cooperative, 
hard working, concerned and dedicated staff. 

I began a series of informal chats with a variety of teachers 
during which I asked things like, "Hcvy do you decide what skills to 
teach?" "What does introduced mean to you?" "Do you think you can 
cover all ski^s required for your level?" "Which skills are most 
important to your grade/class?" - 

By listening carefully and reading between the lines a bit, it be- 
came increasingly clear that my group of dedicated upper grade teachers 
were having a terrible time. The complaints and concerns boiled down 
to the fact that for many of them it was the first time they were being 
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forced to teach reading, language and math skills and they weren't any 
too sure what these skills were and how to teach them* So long as they 
followed the textbopk, they were fine. All of a sudden they had skills 
to teach and no reference to a specific page number in a book. For 
many, particularly in reading, it was the first time they were con- 
fronted with subdivided content skills* The component parts of reading 
were all of a sudden isolated for them and many had never taught these 
component parts. 

We realised that in order to get back on track and make the 
management system an effective tool for us, we needed to do several 
things. We needed to: 

- narrow the laundry list of skills contained on the continua 
so that teachers would learn to teach skills in digestible 
hunks; 

- select skills for this narrowed list, that were critical to 
mastery of each subject; 

.- select skills that were consistently critical across the 
three grades we serve: 

- select skills that were measured on most of our assessment 
instruments; 

- help teachers value these skills; 

- help teachers learn how to teach these skills. 



The steps we took to accomplish these tasks were to: 

- use standard reading, language and math committees to discuss 
and isolate 10-15 skills that were critical and that were 
necessary at each ghade level; 

- select reading as our focus for spring 1982 with language and 
math to follow in the 1982-83 school year; 

- cross-match skills selected by the reading committee to the 
test specifications for Hart Bill, CTBS, and CAP; 

- select the 10 skills most commonly measured and critical to 
mastery of reading; 

- divide the staff into three committees with each being 
responsible for: 

° refining the district's task analyses for each of 3-4 
skills; 
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° developing additional CRT's for grade levels where none 
were available for each skill; 

0 developing/ordering/adopting teacher support material for 
each ski 1 1 ; 

° sharing task analysis content, CRT's and materials for 
each skill with the rest of the staff; 

° identifying staff resource persons for each skill. 

We were able to refine the task analyses, develop additional CRT's 
and order materials by the end of the year. Sharing the task analyses 
and identifying resource persons v are agenda items for September 1982 
faculty meetings. We will then begin the process again for language 
in October of 1982. Teachers will be required to begin to complete 
profile cards for reading in September. As we complete language and 
then math in the spring, they will begin to complete cards in these 
areas. 

One of the terrific things that happened in this process is that 
by narrowing our focus to critical, often measured skills, teachers 
were able to focus on only a few skills to learn how to teach at 
once. They own the program now because they have developed it and 
taught it to each other. Their confidence is high and they 6 ve 
painlessly learned new skills. Next steps, of course, include 
expansion of skills to be addressed and extension of resources and 
teacher skil Is. 

One critical point was that by cross-matching the skills we se- 
lected with the instruments we use to measure skills (CTBS, CAP, Hart 
Bill), the skills were validated for teachers and I can be sure our 
students are being taught' the skills on which they will be tested. 



Staff Development 

Curriculum development and staff development go hand in hand. Curricu- 
lum development js_ staff development. The teachers, principals and central 
office staff members who designed, developed and are implementing the curri- 
culum management system were definitely engaged in orofessional develop- 
ment. The activities described in this section are specifically the staff 
development activities that occurred at the district Professional Develop- 
ment Center. * 
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The Vallejo Professional Development Center was established in 1976 
with state funds and by law served only the 10 Ti+le I elementary schools. 
In 1978 federal Teacher Center funds allowed the Center to serve all of the 

J 

district's schools. The district staff development program was used to help 
train teachers an»J principals to make effective use of test data ^d to 
improve instruction. 

The Professional Development Center program identified specific skills 

r 

areas tc feature !n its task analysis/lesson design training sessions, 
"Identifying the main idea," "inference," "differentiating fact from 
opinion*" were among the skills areas the district identified for focus. 
The t*sk was essentially to improve the job being done in district class- 
rooms in teaching these skills. w > 

Once teachers received training, they received follow-up visits iri 
their classrooms. Principals, trainers and other teachers observed the 
teacher putting new skills into practice in the classroom and provided 
feedback. The follow-up component of the training assured that new skills 
would be applied and promoted ongoing teamwork and dialogue about effective 
i nstruction. 

Improved Testing Procedures 

In addition to curriculum development and staff development efforts, 
procedures were instituted to improve actual testing procedures. A coordi- 
nator was appointed to organize the testing program and to work with school 
sites to implement it. Li addition to CAP and CTBS testing, a major task 
was to develop, pilot and adopt proficiency tests. Because of all the work 
that preceded proficiency testing movement, the district was prepared to 
address the new challenge in a systematic way. 
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The development of the district continua was described earlier. Subse- 
quent to continua development, three teachers were released from their se- 
condary teaching assignments to help other teachers incorporate the skills 
on the continua and tested by the proficiency test into their courses. The 
use of experienced classroom teachers to help other classroom teachers 
proved effective. 1 

Principals' meetings were also used to improve testing procedures. At 
key points during the year, the importance of testing was reviewed with 
principals. They were asked to come to meetings prepared to share strate- 
gies that Worked for them to help students and teachers take testing seri- 
ously, to schedule testing at times when there would be a minimum number of 
interruptions and to use test results in a planful way. Principals learned 
many valuable strategies from one another. The superintendent chaired the 
meeting as he does most principals' meetings. His clear direction that ef- 
fective testing procedures were important re-emphasized a district priority. 

W here We Are Now 

The directions set as a result of the initial concern about testing 
continue. Curriculum development, staff development and improved testing 
procedures remain priorities. The past six years marked a period of very 
acive development. The current thrust is to thoroughly implement and re- 
fine what exists. 

A great deal c* energy went into the developmental phase of the curri- 
culum products. The current thrust is to see that the products are used 
rather than shelved. In an earlier vignette, one principal described 
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the difficulties of actually using CRT's and profile cards. It is a slow, 
painful process. Each principal submits annually an objective to the^super- 
intendent for providing the leadership for his or her staff to proceed to 
the next level of use of the system. The Curriculum Director provides di- 
rect assistance to principals who are evaluated by the superintendent on the 
degree to which they accomplish their objectives. 

The staff development program will include more and more training ses- 
sions on how to apply effective instructional practices to using the dis- 
trict's curriculum products. The Staff Development Department and the Cur- 
riculum Development Department are working together to find teachers who 
have worked out solutions to typical curriculum development problems who can 
train other teachers. The follow-up component described earlier continues 
to emphasize and assist in the application of new skills in classroom 
instruction. 

Testing procedures continue to be improved both through the central 

coordination of efforts by the Coordinator of Research and Assessment and 

/ 

through the site leadership of principals. Principals' meetings continue to 
be used to help train principals and maintain a focus on the effective use 
of test results for instructional improvement. Without involving school 
staffs, a curriculum may only be words printed on pages. In order for a 
curriculum to be taught, it must be wrestled with. Teamwork is an essential 
ingredient of success and teamwork, too, takes time. A real success is the 
current level of teamwork. 

Planning is another key ingredient of success. An extensive effort to 
revalidate district goals with school staffs, students, parents, and the ^ 
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community was recently developed. Without a long-range plan and a timeline, 
it is too easy to drift away from directions. The plan was developed with 
input from' many teachers and principals. It will be presented to all school 
staffs, parents- and the public. An Instructional. Improvement Council con- 
sisting of administrators, teachers, classified employees, parents, stu- 
dents, and community members will monitor the plan. 

A final success has to do with morale. Teachers and principals are 
probably working harder than ever before. They are tackling serious issues 
in a serious way. The teamwork, sense of purpose, and sense of accomplish- 
ment has had a positive effect on morale. The danger always remains that 
dedicated professionals will work too hard, too fast, and too long and burn 
themselves out. The purpose of the five-year plan is to maintain the vision 
and intensity but pace the efforts in a simple and satisfying way. 

Lessons Learned 

Many lessons have been learned in Vallejo. -Experience has taught that 
the most important endeavors require a great deal of time and commitment. 
Our experience has shown that there are some possible pitfalls, some ques- 
tions that are frequently asked, and some successes to share. 
Poss ible Pitfalls 

— /> 

There are pitfalls to avoid. Our experience would lead us to caution 

other districts not to attempt a comprehensive improvement effort unless 
they are willing to allocate the necessary time and human and finanical 
resources, 
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The greatest cost is staff time. In Vallejo, the Administrator for 
Instructional Services, the Curriculum Development Director and the 
Professional Development Director have allocated a substantial amount of 
£heir time to the efforts described here. Release time has also been a sub- 
stantial expense both for curriculum development and training activities. 
The time period in which Vallejo's efforts were initiated was one in which 
state and federal grants^ were more readily available. The district's com- 
mitment was essential but'some aggressive grant writing efforts also (gelded 
some financial resources that probably allowed the efforts to develop more 
fully in a shorter period of time than would have been possible with no 
external sources of funds. A real pitfall would be to undertake a serious 
change effort without examining the costs. 
Questions Others Ask 

We are often asked about the balance between leadership and involve- 
ment. What we have found is that leadership and structure is appreciated as 
long as clear and relevant opportunities for involvement are built into %he 
process. The superintendent and his staff are very much in charge in-^erms 
of establishing directions, but school-site staffs have the flexibility to 

tailor district directions and programs to their needs. Both the curriculum 
§ 

development and staff development efforts have built in a wide variety of 
ways to be involved as well as varying degrees of involvement. Some teach- 
ers and principals are involved only in school -site efforts while others sit 
on the Instructional Improvement Council. Some teachers and principals 
choose mainly to receive and apply training in their own job situation, whil e 
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others stretch themselves to receive and apply training and then train 
others. There are many avenues for individuals and groups to participate in 
the instructional improvement efforts. 
Factors Contributing to Success 

The importance of district level leadership is constantly revalidated., 
The superintendent sets clear priorities, evaluates principals on their 
ability to carry out those priorities, and directs his staff members to use 
their resources to assist- the school sites in their efforts to address the 
priorities. The superintendent's vision is .an important prerequisite. 

A second factor contributing to our success has been the willingness to 
take the time needed to accomplish change. A curriculum management system 
takes a long time to develop and even longer to implement. Involvement 
takes time, but it is time worth spending. 



\ 
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THE SAN JUAN EXPERIENCE: CREATING AN EFFECTIVE 
EVALUATION/PLANNING MODEL 



by Larry Crabbe 



Introduction 



The areas of program evaluation and planning are relatively new to edu- 
cation. Much of the attention given to them in recent years has come as a 
result of legislated requirements attached to such specially funded programs 
as California's Early Childhood Education (ECE) and School Improvement 
Programs (SIP) as well as the indomitable ESEA Title I Program (now ECIA, 
Chapter I). * 

Ten years ago, the San Juan Unified School District created the posi- 
tion of Evaluation Specialist specifically to carry out these mandated eval- 
uation and planning responsibilities. This was the beginning of an exciting 
experiment which has produced dramatic changes (improvements, we hope!) in 
the ways in which our schools' manage their educational affairs. 

Our evaluation/planning model is one which takes place "where the 
action is," at the school-site level. It also involves those with high 
stakes in its outcome, namely, the school's staff, students and parents. 
Significantly,' our model is one which begins with a broad data base. Most 
importantVy* it goes on serving throughout the school year, long after the 
formal process of planning has been concluded. \ 



In the early seventies, our model was used in fewer than a dozen dis- 
trict elementary schools. As the number of programs expanded, so did the 
application of our model. It is now in -Effect in over 40 of our district's 
elementary, intermediate and secondary schools. We are proud that all of 



158 



- 154 r 



our evaluation and planning support activities, and, in fact, many addi- 
tional Sctivites, have been provided to all these schools with no addition 
in staff (a teacher and myself) from the days when less than a dozen were 
involved. In fact, our entire Research and Evaluation Department has, 
throughout its history, remained exceedingly small with a specialist staff 
of four and a budget representing a fraction of one percent of the dis- 
trict's overall budget. 

The most important outcome from the evaluation/planning process has 
been instructional programs which are tailored to the actual needs of the 

» 

students. These programs are "owned" by teachers and, therefore, are sup- 
ported by those charged with their implementation. The most important by- 
products at the school -site level have been a strong sense of school -commu- 
nity relationship, as well as feelings of openness and self-confidence. 

In the succeeding pages I will describe our moWl . I will indicate its 
philosophical basis, its major components and its chronological evolution. 
It will be helpful, I think, to "set the stage" by discussing the historic 
and current contexts in which our. evaluation/planning model has spent its 
youth. 

< 

Historic and Current Context 

When James Marshall discovered gold at Sutter's Mill in 1848, 
Sacramento, California consisted only of Sutter's small fortress. That was 
all The infamous gold rush that followed i nstantly transformed Sacramento 
into a thriving city of considerable size. Many of the other towns^tojwhich 
that great rush gave birth slowly faded with the demise of the rush. How- 
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ever, Sacramento, because of its strategic location at the juncture of the 
two largest rivers of the great central valley, continued to prosper and 
grow. This trend has continued, nearly uninterrupted, to the present day. 

As Sacramento grew, small communities began to dot the surrounding 
countryside. As the years' passed, they formed their own school districts to 
attend to the educational needs of their children. Six such school dis- 
tricts, twenty-two years ago, combined to form the San Juan Unified School 
District. With the passage of time, these individual communities and their 
district offices merged to form the north-eastern suburban area of 
Sacramento. 

San Juan is now the seventh largest school district in California. It 
serves over 44,000 kindergarten through twelfth grade students and spreads 
its influence over an area of approximately 79 square miles. The district 
as a whole has been described as "middle" to "upper-middle 11 class. Socio- 
economically, San Juan exceeds 80 to 85 percent of California's unified 
school districts. Sacramento, the capital city of California, owes much of 
its economic existence to federal, state and local government as well as to 
the military and aerospace. These "industries" utilize the services of many 
professional and paraprofessional employees. After work, many of these 
people come home to the vast "bedroom" community served by the school 
district. 

As might be expected, the district's students have strong academic 
orientations. Student performance, according to the state's testing pro- 
gram, is in the 73rd to 87th percentile range. Three out of every four 
graduates attend college. There has been a gradual increase ir racial and 
socioeconomic variety. The current racial composition is 1.9 percent 
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American Indian, 2.5 percent Asian, 1.9 percent Black, 4.6 percent Hispanic 
and 89.1 percent Caucasian. The percentage of students receiving public 
assistance (AFDC) varies widely from school to school, from a low of less 
than one percent to a high in excess of 40 percent. 

In recent years, the district has faced several significant chal- 
lenges. Our most enduring problem has been financial. Because of our 
suburban character, the district's tax base and, hence, resulting revenues 
have been and continue to be low. The community's support of public edu- 
cation has resulted in a relatively high tax effort. 'But despite this, the 
district's expenditures still average $30 to $150 less per pupil than the 
median district in the state. This situation is now complicated by the 
state of the economy and the discontinuation of funds for special programs. 
District costs are escalating while, at the same time, funds are drying up. 

A second recent challenge to the district revolved around its board of 
education. Our traditionaly peaceful district has in the past several years 
received much statewide attention as a result of, not one but two board 
recall elections. 

The third challenge is declining and shifting enrollment. Like many in 
the state, we are experiencing a gradual decline in enrollment. Were this 
not enough, our district situation is compounded by a shift of students 
from west to east within our long and narrow district. As a consequence, 
schools in the older western portion of the district have surplus capacity 
while those at the opposite, eastern end are bursting their seams. Solu- 
tions to such problems, especially in districts of this size and wealth, do 
not come easily. 

1 81 
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This, the^ v ^vs the context in which we labored to develop our approach 
to educational program evaluation and planning. 

Our Philosophy 

I mentioned earlier that the district responded to early legal evalua- 
tion and planning mandates by creating a position, Evaluation Specialist, 
with those mandated activities as its job description. This decision, pos- 
sibly more than any other, may have been responsible for the success that 
the district has had in the area. This may have less to do with the indivi- 
dual filling the position, me, than it does with the fact that a position 
was created with evaluation and planning as its direct responsibilities. We 
all, 'quite reasonably, concentrate the majority of our efforts in those 
areas upon which our continued employment most directly depends. We also, 
quite reasonably, limit the amount of time, we devote to peripheral duties so 
as to ensure that our primary responsibilities are not neglected. 
Purpose of Evaluation: Planning 

Throughout the evolution of our evaluation/planning model, we have held 
true to several fundamental beliefs as to tho reasons for evaluation and 
planning. We believed that the evaluation and planning model would deliver 
the greatest "payoff" to the district's schools if these were its primary 
goals: 

° to improve the quality of the local educational program; 

° to encourage the most effective or efficient use of scarce 
financial resources; 

° to assist with the attainment of the goals of the school, the 
school district and the special program(s) mandating the 
evaluation/planning activities. 
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There was another, secondary, goal for evaluation and planning: 

° to create a written document or "plan" in which to preserve the 
most significant findings or conclusions emanating from the 
evaluation/planning process. 

r 

If well attended to, this secondary concern can assist schools signi- 
ficantly to move toward attainment of our primary goals, even though it 
should never, never, exist as a goal or end in itself. A well written plan 
or "grant" can certainly have a favorable financial impact on the school or 
district. But, an impressively written grant does not ensure that the 
resulting funds will be used effectively. Thert; is a vast chasm between 
true planning and "grantsmanshij)." Schools are definitely shortchanged to 
the extent that the evaluation and planning involved in the grantwriting 
exercise are "simulated" rather than "real." But if the evaluation/planning 
effort is real , a wel 1 -devel oped written plan format can be of immense value 
in insuring that the fruits of planning are not lost but remain 'avail able 
' for use during the implementation phase. 

You will notice we talk about the concepts of evaluation and planning 
as inseparable companions. It is our belief that the value accruing to the 
school district relates very directly to the closeness of the linkage be- 
tween the two. In fact, we believe that the major reason for evaluation. is, 
in fact, planning. We believe that evaluation should exist to enlighten 
decision makers. 

The close relationship between evaluation and planning depends upon an 
equally close relationship between evaluator and planner. Within our dis- 
trict, this problem is easily solved: they "are one and the same person. The 
Evaluation Specialist is also the banning Specialist, though not by name. 
To some evaluators this may be a scandalous situation. It is, therefore, 
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important that you understand our position. In our district, most evalua- 
tion and planning activities take place at the school site. The Evaluation 
Specialist acts primarily as a resource person and facilitator. Most of the 
actual evaluation and planning "conclusions' 1 are reached by those participa- 
ting at the local level. The Evaluation Specialist is the developer and 
custodian of the evaluation and planning processes. Those at the local 
level are the actual participants. The Evaluation Specialist is also the 
manufacturer and provider of data;* Those involved at the school level are 
the actual consumers. For these reasons, much of the traditional concern 
for evaluator independence is unnecessary. While the Evaluation Specialist 
could certainly be expected to be biased with regard to the value of* the 
processes involved, he does not necessarily have to be biased with regard to 
the program which results at any particular school site. What this dual 
role does do is prevent the impotence which often results when the evaluator 
is absent during the critical latter half of the evaluation/planning 
process. 

Our Planning/Evaluation Cycle 

We have, thus far, discussed our philosophy as to the "why's" of evalu- 
ation and planning. We have thoughts regarding the "how's* 1 of evaluation 
and planning. 
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First, any evaluation/planning cycle should parallel the essential 
steps of human problem solving which are. . . 



RECOGNIZING WHAT 
PROBLEMS EXIST 



RECOGNIZING THE 
ALTERNATIVES 



RECOGNIZING WHY 
THEY EXIST 



IMPLEMENTING 
ALTERNATIVES 



<-l 



SELECTING FROM 
THE ALTERNATIVES 



Our evaluation/planning process acts as our "tour guide" leading us 
through each stage of problem solving. 

Our evaluation/planning process is comprehensive, in the same way a 
physical examination is comprehensive. The medical doctor cannot affort to 
overlook a single area of medical significance to his patient. Nor can we, 
as educators, afford to overlook any of the educational essentials if we 
wish to maximize our students' educational wellbeing. 

Our evaluation/planning process is "broadbased." It involves all ele- 
ments of the school community. This includes students (at the proper level 
of maturity), parents, staff members as well as administrators.- The value 
of the varying perceptions and points of view which exist within the varioi 3 
groups should not be underestimated. Also, being participants in the pro- 
cesses of evaluating and'plannirtg can- develop a deep sense of commitment 
toward the resulting program as well as a kind of "bonding" between school 
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and community. This participatory aspect of evaluation and planning places 
the site administrators in a unique and key role. 

Our evaluation/pi .nning process is "data-based. 4 Le mentioned that we 
believed it was evaluation's primary responsibility to erjrfighten planners or 
decision makers. One means of enlightenment is through available informa- 
tion regarding student performance, attitudes, etc. Vie have contacted end 

"heard from" each student through the medium of a written instrument. We 

t 

would seldom, if ever, have the time to do this in any mor? literal w?y. 
So, surveys and the like serve us by gathering those student, parent and 
staff perceptions which we acclaimed earlier. The more such information* we 
have, the better our decisions are lively to be. Our educational program 
and our students are the ultimate beneficiaries. 

The Evolution of Our Model 

One has to reflect back over ten or f i rteen years to re:ognize the r'ra- 
matic changes which have occurred in the way we operate our schools in the 
area of evaluation and planning. 
Initial Resistance 

At the beginning, my feeling was one of loneliness, like being out on a 
limb all by yourself. The attitude in our schools regarding "outsiders" — 
not only local visitors, but also staff members and administrators from 
other schools ~ was often pleasant but guarded. The feeling toward cen- 
tral administrators was often one of privacy, protection or suspiciousness. 
When we began getting down to business, the atmosphere frequently became 
even less pleasant. And the dif ferences^ were not limited to attitudes. 
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The aversion to what might be described as program management activi- 
ties seemed widespread: "Principals are teachers' teachers, not managers!" 
Since formalized planning was much less frequent than it is today, programs 
were based on enduring, but perhaps obsolete, assumptions rather than on any 
assessment of participant needs. Programs were incompletely planned, with 
significant areas regarding objectives and curriculum omitted. There was 
only a limited sense of accountability for program funds regarding such pro- 
grams as ESEA Title I. Staff commitment to such programs was limited by a 
general feeling that these programs had bean "laid on." Lastly ^-but very 
significantly, there was limited parental' support for such programs fre- 
quently accompanied by criticism and suspicion. 

I mentioned that staffs felt a limited sense of commitment and also 
that programs were "laid on." Such was exactly the case, and so specified 
by state regulations. At that time, districts throughout the state cen- 

J! 

trally developed a'single Title I plan and then shipped it out to the pro- 
ject schools for implementation. The program contained objectives which 
served as the basis for a single central program evaluation. These objec- 
tives were not considered by the schools to be their objectives. 

It was this condition that provided the impetus fo.^ change to the 
si\.e-level approach we see in -use today. 

T^-i years ago, when our Evaluation Specialist position was established, 
the district had six elementary schools in its Title I program. School per- 
sonnel continually ab.ced that they be allowed to create the* r own program 
pans suiteo to the uniquenesses of their individual school sites. Serious 
doubt was often expressed by federal or state administrators with regard to 
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the ability of local school sites to accomplish such a task. However, in 
1971, the legislature passed legislation creating the Early Childhood Educa- 
tion (ECE) program and, with it, a framework for evaluation and planning at 
the school level. The California State Department of Education developed 
the evaluation/planning model as well as the planning format. The creation 
of the ECE program prompted us to develop a participatory evaluation/plan- 
ning model based on a broadbased needs assessment process'. Our efforts were 
shaped by the state's process and by its forms. 

Before discussing what we did, I think it would be worthwhile to dis- 
cuss the atmosphere within the "central office" toward our efforts. Early 
on, central administrative awareness, understanding and appreciation of our 
* efforts were limited as was their support. There were several highly placed 
strategic exceptions, namely, the Director of Research and Evaluation and 
his superior, the Assistant Superintendent for Special Services. They have 
always philosophically supported the importance of educational evaluation. 

We also had a critical ally in the ECE/Title I Director. He had based 
his doctoral dissertation on community needs assessment. Many times, he and 
I shared the feelings of isolation which were so much a part of those early 
days. At one point, he asked me if I realized that it was just he and I 
trying to convince a skeptical district of the value of categorical pro- 
grams, planning, evaluation and the like. I will never forget the 
"friendly" weekend mountain retreat for project principals which evolved 
into a kind of marathon Di rector/Eval uator "roast" lasting two days. 

We. knew that for our efforts to be successful we would have to win the 
central and field administration over to our point of view. We attempted to 
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accomplish this by deed. We know that evaluation and planning^ activities do 
create a burden for teachers and principals. Our evaluation/planning acti- 
vities had to solve more problems than they created. The values of our pro- 
cesses had to be made vi sable to students and management. We successfully 
used our model several times to facilitate the resolution of school -commu- 
nity conflicts in schools outside the small group of. project schools. We 
can now report that our relationship with both central and field administra- 
tors is an exceedingly positive one. It is satisfying to se'e, and espe- 
cially to feel, the contrast. k . 
Needs Assessment Survey 

Let's return now to our needs assessment process. Our earliest ver- 
sions of the staff, parent and student survey instruments came to be widely 
known as the Educational Program Assessment (EPA) process. Since its crea- 
tion, the ongoing work on the EPA instruments has involved large numbers of 
parents, staff, administrators and secondary students. This has given us a 
unique opportunity to test out a theory regarding the beneficial effects of 
involvement on school and community feelings of ownership and support for 
such processes. 

We believe that broad school and community involvement has contributed 
to responsive instruments. These, in turn, have led to high response rates, 
averaging about 70 percent and rising to an unbelievable high of 100 percent 
response from one school and its community. 

There were, I believe, several additional key elements in our suc- 
cess. First, our surveys were very unusual in that they, unlike many others 
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conducted in the educational setting, were done "by 11 the school and "for" 
the school. It' was one of those rare i nstances where information actually 
completed the loop back to its point of origin. School people were used to 
seeing a survey when filling it out and never again. This time the result- 
ing data was returned to them for their use, a very si gr^f/cant difference. 

While school-site involvement in instrument development and completion 
of the information loop were necessary, they were not sufficient to ensure 
the positive effect we p wanted from our efforts. We carefully considered the 
content, the format and the length of the instruments themselves as wel 1 as 
to the manner in which the resulting data were presented for use at the 
school level. 

The questions in our survey instruments addressed not only those areas 
of particular interest to the sponsoring special program(s), but also those 
areas of general concern to the entire educational program. In this way, 
much of what a school wanted to know from its community could be obtained 
from a single survey. Survey questions asked about parents' perceptions of 
the effectiveness of the current program in various areas of the curriculum 
as well as the need for improvement in their child's performance. Questions 
were limited to those which respondents could reasonably and intelligently 
answer. For example, parents were asked about the performance attitudes, 
etc-.— of- the4^-own-cfvi 1 dren- and not those of other students i n the ' school or 
neighborhood. 

As I mentioned, we were concerned about the way in which the data was 
delivered back to the school. Here, we ran into a ailemma. Popular at thzt 
time was a belief that broad participation in any activity, including survey 
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tallying, was beneficial. This is precisely how we formatted our surveys 
during the first year. This was, in part, to test our belief in participa- 
tion but mainly it was practical. There was no other way of doing it at the 
time. All went well, but it became apparent to me that the cost, in terms 
of tine, accjracy and effort, exceeded the actual benefits. I came to be- 
lieve that the greatest benefit accrues from easy access to the resulting 
information, not from the process used to analyze it. t 

The second set of arrangements made for survey processing was even more 
interesting. We used the computer facilities of a university in a neighbor- 
ing community, having keypunched the surveys within the district. The data 
decks along with the control cards for the statistical ■ software being used 
had to be physically transported to the university computer center. This I 
■■■ -illy ■'id on my way home fr.sm work. The next morning, I retrieved the 
.;?cks and printouts or, my v jy to work where a team of clerical workers tran- 
.s;ribed their contents onto a specially developed graphic format. In spite 
C its compexity, ttm arrangement was a definite improvement over hand 
tallying. 

But finally, - moved into a third phase for survey processing. My 
commuting back >rd fs-th between' the district and the university for one 
year provided tit ■ .11 strict with t\y> incentive to develop our own computer 
program for use~within the district. We then could tabulate the results of 
each survey ir. both graphic and" tabular form. 
Standardized .ests 

While survey responses do provide a large amount of information for 
evalu?tion and planning, they do not provide it all. The primary source of 
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our evidence about student achievement is the standardized test, in our case 
the Iowa Test of Basic Skills (ITBS). We needed to maximize- the value and 
validity of the test information while/minimizing its cost in time and ef- 
fort. Title I regulations had mandated fall and spring testing which seemed 
an enormous amount of testing with its predictable negative effect upon the 
students and staff. Calculating academic gains between fall and spring, as 
was the practice, brought with it all sorts of Complications and excluded 
the significant effects of the summer period. Ironically, due to this 
spring testing requirement, the most useful information for local evaluation 
*rvd planning typically became available to schools at or after the end of 
the school year. For these reasons, the districts in the Sacramento area 
requested, and were granted, a waiver so that they could implement an annual 
(spring-to-spring) testing schedule. The gains from such testing cumulate 
over a number of years and yield a more accurate reflection of student 
learning. Perhaps the greatest benefit was that the information for evalua- 
tion and planning was now available at the outset of the next school year. 

While we had overcome a major problem in the area of standardized test- 
ing, another related major problem went unattended until very recently. 
This had to do with the form in which standardized test results were pro- 
vited to schools. This was not a problem unique to the San Juan Unified 
School District but can be found coast to coast. District offices, as do 
the scoring services offered by test publishers, ordinarily return results 
as computer printouts, usually perceived by teachers as unattractive and 
technical. However, this is only half of the problem. The other half has 
to do with the staggering number of such printouts which must be consulted 
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in order to draw the kinds of conclusions that are critical to evaluation 
and planning. 

For example, a test with four subtest areas will require 24 printouts 
to summarize a single year's results (grades 1-6). The typical Title I 
school will also want separate results for participating students, requiring 
another 24 printouts. Much of the value from standardized test results 
comes from the examination of results over a number of years. Looking at a 
four-year period for these two groups requires 192 pages of printouts. For 
a separate report of gains for each of these groups an additional 120 pages 
is needed. That is a grand total of 312 pages, more than enough to intimi- 
date the most Ambitious administrator. 

This problem led to the development of our Academic Performance Profile 
(APP). Each Academic Performance Profile contained graphic and statistic 
results for a particular subtest area. On this single profile are the re- 
sults for "all the grades" for "all the years." A set of eight profiles 
contains the information formerly contained in 312 pages. 

Another of our conquests is a major advance in the evaluation/planning 
process,, and deals with the' actual act of budgeting. Formerly, it was left 
until after planning was complete. But, our recent modification to the 
planning format has created a proper place for budgeting with in the evalu- 
ation/planning process. Each activity in the program description is now 
linked to the funds and the funding source which makes it possible. An 
easy-to-read "audit trail" has thus been created. 

An evolutionary process has, thus, brought us to a point where we have 
a needs assessment process, and an efficient and effective means by which to 
present standardized test results. 
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A Description of Our Model 

Any discussion of our current evaluation/planning model should focus on 
four major areas: our human evaluation/planning activities at the school 
site,, our program plan forms, the information gathering and organizing 
processes and- our district level evaluation/planning activities, j 
Evaluation and Planning at Schools 

j As we mentioned earlier, the evaluation/planning model is systematic 
problem solving, i.e., identifying existing problems, isolating the 
mental reasons for those problems, discovering and evaluating alternative 
solutions, adopting the most promising solutions, and going about our busi- 
ness. Our evaluation/planning process takes people through six steps which 
gets them to problem solve at the school -community level. 

Evaluation and planning activities at the school take place under the 
patronage of the School iite Council, a fcody whose composition and influence 
has grown out of the California School Improvement Program (SIP) legisla- 
tion. This committee, with the site administrator, represents each of the 
major constituencies within school -community (i.e., parents, s$aff, admini- 
strators, and secondary students). 

The school 's curriculum is typically (although not always) separated 
into academic components such as r ead ing, language , mathematics , etc . The 
School Site Council usually creates "component committees 11 of parents, staff 
members and students (in secondary schools) to do evaluation and planning 
for each academic area. The final school program plan will have sections 
addressing each component written by the separate committees. 
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At the outset, each component commitee tries to define their area of 
the curriculum comprehensively, to ensure that all sub-areas are addressed 
in the evaluation/planning process. 

The committees of educational planners, like physicians, begin by 
determining the status of their students' educational "health" in each 
significant area. 

When a physician's examination and tests are complete, his or her job 

i 

has only just begun. The same is true for the component committees. The 
members of the component committees examine their newly gathered informa- 
tion to identify the strengths and weaknesses of their students. With this 
step complete, the component committees turn their attention to the existing 
programs. In those areas where student performance has been found to be 
unsatisfactory, an effort is made to identify probable' program causes, that * 
is, program gaps or weaknesses which, if eliminated, would result in 
improved student performance. 

i 

Once a doctor has completed his review of the existing treatment plan, 
he is then ready to progress to the presciption phase, in which a new, or 
perhaps a modified, treatment plan is developed. Not unlike the physician, 
educators have the responsibility of remaining aware of the variety oriftra- 
tegies and materials which are potentially available to meet the needs of 
their "clients". School people join with others who have special skills in 
the curricular areas and are available for consultation. After such con- 
sultation, members can modify, extend and improve their school's existing 
program. When they have finished deliberating, they use the Program Plan 
form to document their new program description. 
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Were schools not constrained by budget and time limitations, it would 
be simple to adopt all of the modifications and additions proposed by the 
various component committees. However, dollars and time are definitely 
limited. Therefore, there must be a forum to make these inevitable time and 
financial budgeting decisions. We return to the School Site Council who 
does a component-by-component review of the proposed program after an anti- 
cipated projected cost has been attached to each proposed expenditure. 
Through negotiation and compromise, the activities of lowest priority, along 
with their related costs, will be reduced or eliminated to the point where 
the amount which the school proposes to spend wilT^exactly equal the amount 
which it expects to receive. 

There is something of great importance which occurs at this critical 
juncture in the evaluation/planning process. Up to this point all activi- 
ties have occurred in the individual component committees. In order for 
ownership and commitment to be strengthened there must be this meeting of 
the members of the total planning group. The ultimate result, sometimes 
after many emotional hours, is a program which is widely owned and 
supported. 

When the doctor has completed his or her diagnostic activities and 
prescribed a course of treatment, h£ is often in a position to offer an 
opinion regarding the probable effects of the treatment plan on the future 
health of the patient. This is called a "prognosis." Our schools accomplish 
this step when they identify their program objectives and determine a means 
of measurement for each. It is our belief that the greatest value of 
objectives does not come from identifying them but rather from the data 
gathering activities which they stimulate. It is also our belief that 
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objectives become much less fanciful when they are not prematurely 

formulated. 

Program Plan Forms 

We have already mentioned that our program plan format was designed so 
that there is a very close relationship between its layout and the various , 
steps of the eval^4tion/planni ng process. The columns are sequentially 
organized from left to right to parallel the steps of the evaluation/ 
planning process. The format assists the experienced planner/evaluator in 
documenting, his/her findings. It functions as a guide for the less 
experienced by presenting the steps in their proper order. 

This format has undergone a number of major revisions. The State's 
first attempt at a form for use was a complicated two-page affair with need- 
less redundancy. The State's willingness to allow districts to experiment, 
with the form itself has led to major improvements. 
Information Gathering and Organization 

We have described the survey instrument on which we gather the views of 
each of the three major segments of the school community parents, staff, 
and secondary students. The survey is composed of a minimum set of standard 
questions common to all schools plus a second set of optional school - 
developed questions unique to each school. Our assistance is provided to 
requesting schools in developing questions. The district summarizes these 
school -develo-ed questions along with the standard questions used at every 
site. 

Each staff member and^secondary student completes an EPA survey form, 
while the parent surv / is completed one-per-household. The responding 
parent is directed to randomly select one child as the survey's focus. 
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One set of "standard" survey questions request that the parent or 
secondary student; assess the effectiveness of the school in teaching the 
student what he/she needs to know in each of fourteen curricular areas 
(eighteen at the secondary level). This set of questions tries to establish 
the status quo in terms of the extent to which the student x % academic needs 
are being met. There is another side to the coin: how much or how little 
need is there for change in the status quo? That is, ho* much need is there 
for improvement in the student's performance in each' of those same areas? 
A second "standard'^set of questions brings student and parental values' into 
play. The fact that a particular student's needs are not being met effec- • 
tively in a particular area of the curriculum does not necessarily mean that 
program improvement is desired either by the parent or the student. The two 
sets of questions must, in our view, be used together to produce a complete 
picture. 

Surveying takes place in the spring of each year. Our office schedules 
evaluation/planning release time. We are aware of when each site's survey- 
ing must begin in order that the tabulated resuHs are available during 
their scheduled release time. The standard sur. forms are printed and 
shipped from the central office. Any desired "optional" school questions 
are added at the school prior to their distribution. The surveys are re- 
turned to the central office for processing. 

Earlier we described the exhausti ve-efforts that have gone into the 

presentation of survey and other results. Currently, the data from EPA sur- 

t 

vey instruments are keypunched directly onto tape which is then formulated' 
into tabular and histogram formats. The survey uses fqur-pt?«r?t "forced 
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choice" questions in order to enable the creation of a simple visual index 
of, for example, "effectiveness." Almost instantly, the strongest and 
weakest areas can be identified by consulting the histograms provided. 

While the student and parent survey instruments focus almost entirely 
upon student performance, the staff survey asks for comment on the strengths 
and weaknesses of key educa* activities. This has been most exten- 

sively, developed in the s<»co school version of the EPA staff survey 

where clusters of questions address topics such as the effectiveness of the 
instructional processes, the effectiveness of the curriculum content and 
organization, the effectiveness of the class and school environments in 
which learning takes place, the effectiveness of staff development, the 
effectiveness of guidance and health services,- Responses to each of these 
questions are summarized and presented on a school -wide departmental basis. 
We are developing similar questions for the elementary staff survey instru- 
ment. The same computer program is used to summarize these items as well as 
the other sections of the staff survey. It should be emphasized that this 
computer program was written so as to be useful in summarizing a wide vari- 
ety of local evaluation instruments. 

While the surveys which make up the EPA are a valuable source of evalu- 
ation and planning information, they by no means provide all of the neces- 
sary data. Academic performance in our district is assessed by the Iowa 
Test of Basic Skills (ITf$S). One major product of the recent past was the 
development oi an economical and concise method of storing and presenting 
the results of the ITBS. The method .devel oped was the Academic Performance 
Profile (APP). Th<: APP conveys a large amount of information relating to 
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the academic performance of students assessed by the ITBS. Admittedly, it 
has a complicated appearance at first glance. However, once understood, it 
is of great vMue in enabling tfie reader to master a lot of information in a 
minimum amount of time. 

Juan Unified School District annually assesses the academic perfor- 
mance of its elementary and intermediate students during the month of 
January. Our profile format relies on grade equivalent scores, although the 
format can ^asily be modified fo r use with percentile scores. 

Each profile corresponds to a s^gle ITBS subtest. All ITBS grade 
equivalent information pertaining to the particular subtest is contained in 
that profile regardless of the grade or year. The basic information con- 
tained in the Profile is displayed in two ways: lj in statistical fashion 
in tables situated down the right side of thfe Profile; and 2) in visual or 
graphic fashion in the graphs down the center of the Profile. 

A key table located at the upper right corner of the Profile contains 
grade placement scores for each grar^ and year. These scores can be exa- 
mined by grade level (horizontally) or by student group (diagonally). These 
two types of analyses are shown via the top two graphs in the center of the 
form. Across the bottom of the form, a single graph and tfble present 
year-to-year group gains for the various grades in an unusual manner. The 
gains, are stacked, one atop the other, to produce an aggregate indicator of 
the "total productivity' 1 of the school. 

Within our district, the availability of the ITBS data in this fashion 
has greatly expanded its use. A similar profile has also been developed to 
display the standardized test data for Title I participants only. This 
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profile, in contrast, charts the beginning and ending grade placement scores 
as well as the related grade placement gains for all students identified as 
Title I participants. 
District Level Evaluation and Planning 

Each of oar evaluative instruments as well as their summary displays 
is designed to satisfy district and program evaluation and planning needs. 
Responsive to the informational needs of the board of education, district 
and program administrators, such summaries also serve as valuable frames of m 
reference for those at the local sites. In many instances, these district- 
level summaries are placed on transparencies directly laid over the school's 
summary for easy comparison. 

In all of these activities, our motive has been to create the greatest 
benefit with the least burden. School staff members must be involved in 
program evaluation and planning. They also have a fundamental responsibi- 
lity to be in classrooms, educating students. Therefore, their time out of 
the classroom must be well -utilized. We must see that the benefits to stu- 
dents outweigh the , cost of administrator-arid teacher time. 

/ 

A Look To The Future 

While we are proud of our laccompl i shments t ; date, there are always 
areas in need of attention and imprpvement. We -0 not allow ourselves the 
luxury of feeling totally adequate. We know the, we have a long way to go. 
There are many areas where it still "just doesn't feel right." In some 
instances we know why; in others, we do not. 
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In this age of tenuous financing throughout the educational system, 
perhaps a greater proportion of the potential value from evaluation and 
planning activities could be had if something could be done to consolidate 
special program funding. But this is not the major problem. While the size 
of the check is of, importance, a great deal of difficulty is caused by the 
late date at which districts and, therefore, schools find out what their 
actual funding is going to be for the coming year. Making the most effi- 
cient use of our limited financial resources is one of the most basic goals 
of evaluation and planning. Its attainment is not made any easier when-we 
find out what our budget is only after our planning efforts have been 
concluded. This complication in our life has existed for a long time, 
probably as long as the special program funds have existed. It should be' 
mentioned, because it is one of the factors limiting our success in 
evaluation and planning. 

While we are speaking of budgets and budgeting, there is another situa- 
tion worth mentioning. There is a definite need fo~ inservice training in > 
the area of accounting and bookkeeping for staff members and administrators 
alike. Special program funds and accountability for those funds, not to 
mention the budgeting act itself, have been a source of frustration to those 
inexperienced in such matters. We have only scratched the surface of auto- 
mated expenditure accounting in the management of special program funds. 
The typical district accounting system is not equipped to handle the de- 
tailed budgeting required by special programs. Participatory decision 
making requires up-to-date figures regarding expenditures, unencumbered 
reserves and the like. The application of modern microcomputer and word- 
processing technology holds great promise. 
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Meeting all of the widely varying demands for information at the 
school*, program, and district level is a great challenge. Automation is, I 
believe, the ultimate answer. Few people fully appreciate the magnitude of 
fully automating a district's data needs. Computer programming is a slow 
and arduous process in the beginning, especially when there is intense com- 
petition for computer services. Shrinking funds complicate matters fur- ^ 
ther. We have to look into the future for such advances as an accessible 
district-level .data base. When this is a reality, evaluation and planning 

will realize more of their potential. Theoretically, there are no limits to 

f 

th,-. factors that could be encoded in a data system. Entering a reasonable 
number of "independent variables" could enable us to select one or a combi- 
nation of these variables and examine their longitudinal effects. 

Our goal must continue to be to maximize the benefits while minimizing 
the burdens of any activity outside the classroom, including evaluation and 
planning. While our district has been very' successful in recent years in 
reducing the burden, there is stil room for improvement. We must continue 
to emphasize bona fide planning as opposed to "simulated" planning or plan 
writing. We intend to focus our attention on increasing the depth at which 
we examine our instructional program, concentrating on effective instructive 
strategies. We have come a long way. We still have a long way to go. 
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CINCINNATI'S EXPERIENCE: SUPPORTING COMMUNITY EFFORTS 
TO IMPROVE INSTRUCTION: A FOCUS FOR EVALUATION 

by Joseph Felix 

Introduction 

Improv . instruction despite declining resources is a key challenge 
facing publv; t nation today , especially in urban districts. School 
finance ha? .-cov. .ho crucial issue, debated by educators, politicians, 
parents, ta-... ( -i j and ■*.! -sost everyone else. The financial difficulties 
that have forced many schools to revise their curncular offerings, limit 
2fter-schoo1 activities, and even closo their doors at times are n<Jt~ 
expected to disappear in the near future,. In Tact, with a continuing 
enrollment decline and an aging population with fewer direct ties to 
schools, the picture may even become gloomier. 

For many school districts acrcss the country, budgetary austerity has 
become a way of life. California's Proposition 13 is probably the most 
publicized instance of taxpayer revolt that hes cut into schools' budgets. 
But the Problem is nationwide,, especially among large-city school systems. 

Ohio's requirement that each ir.c'^se of school taxes be approved by 
the voters has made the problem particularly acute for our large cities. 
Columbus, fcr example, has had to make dramatic cutbacks in its research and 
evaluation area, as well in other departments that had .been built to 
notable strength. Cleveland has been on the brink of bankruptcy. 

In Cincinnati, too, the financial problems looin great, A $12 million 
deficit has been projected fo. the 1983 fiscal year. Confronted with such 
fiscal shortage, it is vary difficult to maintain formal efforts to improve 
teaching and l rning. 
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Some years ago I co-authored an article for The ory into Practice on 
Research and Development (R&D) in Cincinnati. I gave the article the 
alliterative title, "Developing Developers: The Race to Improve Education." 
It began: 

Read that title again. Do you hear hbofbcats? 

The race is on . . . For big-city liveries . . .the odds . 
against bringing Lome a winner are astronomical. Under- 
nourished by voters' decree, our horses move forward at a 
pace that would humiliate the tortoise, k seemingly end- 
1 (Oss parade of jockeys mount and dismount with unprece- 
dented frequency. • ■ • 

Cincinnati'? chances of winning the race are prdbably not 
much better than those : of any other rcajor city. But 
we've scouted up an entry fee from ou; benevolent Uncle 
Sam, hired a few stable boys and girls to unstabalize 
thing?., -and mounted a drive for change . . ."(1) 

'WK.i R&D was given its start" through federal funding, there was much 
uncertainty about how long it. would last. "Soft money" was seen as an 
..-round foundation for so important an endaavor. We made efforts to. get R&D 
services mov^d to the "more solid ground" of -local tax money. Now the hard 
g-ound i's ev.en softer than the. soft ground. No wonder our horses stumble 
occasionally! 

Gratefully, I have seen a strong R&D effort survive in Cincinnati 
despite severe budgetary limitations. Credit for this survival must go 
t( ; our Superin • ,,dent, James N. Jacobs, who has national renown as a leader 
in public sc' ; ol research and evaluation. Jacobs had won the confidence of 
the board and much of the staff and community for the cost-effectiveness of 

1. Jacobs, OaleTN., & Felix, Joseph L. "Developing Developers: The Race 
to Improve Education." Theory Into Practice, April, 1972, 11, 225-231. 



evaluation and program development. We continue to believe that a strong 
evaluation unit can lead a school system to making the best possible use of 
its limited resources. 

Recently, we have developed in Cincinnati a major thrust that holds 
considerable promise for further strengthening our instructional program. 
The board and administration have been making a concerted effort to increase 
community involvement in both the decision-making and support processes, I 
want to describe this move to bring the community closer to the schools and 
the measures we have taken to make community involvement the focal point of 
our evaluation effort. 

Despite our budgetary problems, the Cincinnati school system is at an 
historic juncture. Even as local, state, and federal funding decreases, the 
interest of the community in contributing its time and energy for the sup- 
port of schools c6ntinues to rise. The business community has rallied over 
the past five years to furnish us with substantial free consultation and 
support services. A number of local businesses have established partner- . 
ships with individual schools and are contributing both time and money to 
the improvement of these schools. Volunteer service throughout the district 
is increasing in the face of a contrary national trend of decreased volun- 
tary contribution of time. 

In addition, cooperative effort^ have begun with r local universities, 
the City of Cincinnati, and other political and educational groups. These 
endeavors hold considerable promise for filling the gaps caused by limited 
financial resources. 
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What has brought about the school district's success in garnering com- 
r ,iity support? The contributing factors are numerous, but an essential 
ingredient has been the school district's willingness to allow participation 
in the decision process. Our attitude creates, in turn, a very positive 
attitude among the participants. 

Of course, organizations such as the PTA have long contributed time and 
resources to helping us attain scho-ol goals. Their willingness and interest 
in investing in the goals set by the board and administration might have 
dwindled in Cincinnati as elsewhere. But'by inviting parents and other 
community members to participate in planning', our school system is gaining 
commitment to the educational program. We know that the community's will- 
ingness to contribute volunteer resources and to approve tax support is 
increasing. A new organization called Cincinnatians Active To Support- 
Education rallied behind a tax referendum three years ago and gained its 
approval. Community councils are beginning to take an acti.ve interest in 
education, and this often occurs in neighborhoods where other forms of 
pirent and community participation are minimal. 

In 1980, the district started a new program of community involvement. 
This program is a comprehensive, decentralized educational planning process 
carried out by teams representative of each school's total community. Be- 
cause of the district's financial crisis, the initial focus was on budgetary- 
planning. The program was thus designated Local School Budgeting (LSB). ■ 
More recently, the Board of Education passed an action motion calling for 
the establishment of local school advisory committees in all schools. The 
budgeting teams are seen as the prototypes of these .advi sory committees. 
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As with any new programmatic effort, initiating LSB has not been prob- 
leu-free. Community participation in decision making is threatening to some 
administrators. Reaching* consensus on needs and goals is arduous when dif- 

e 

ferent segments of the community have varying priorities. Many questions of 
prerogative and process have to be resolved as the LSB program has moved 
forward. Still, we see the program ; as a significantly successful means of 
improving the education we are providing for the students of our district. 

We have chosen to make the locall school planning effort the focus of 
our evaluation services. We define evaluation in the Cincinnati schools in 
the broad sense of providing information for decision making. For the past 
ten years, the structure and functions of the Evaluation Branch have 
shifted towards serving the local school unit in preference to serving 
central administrators. In the hope that managers in other districts might 
benefit from our experience, I would like to describe how Cincinnati's 
evaluation unit has attempted to respond to the needs of school -community 
participants in the educational planning process. 

To help the reader understand the setting in which we function, let's 
look first at the demography of our school district and the history of its 
evaluation unit. Then we'll discuss Local School Budgeting and the branch 
functions that support it. Finally, we'll talk about the specific role of 
the Local School Evaluator. 

The Cincinnati School District and Our Evaluation Past 
— * 

The Cincinnati school district covers an area of 90 square miles with a 
population of 410,000. The school district's boundaries overlap a number of 
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politically separate areas. In addition, there are within the circumference 
of the district several communities that have their own government and 
school district. 

The enrollment of .the Cincinnati Public Schools presently %qtals ap- 
proximately 51,000. Of these, 57 percent are Black, 42 percent are Cauca- 
sian, and one percent are of other races. The socio-economic level of the 
district is reflected ir) the fact that 56 percent of the enrolled students 
qualify for free or reduced-price lunches. 

There are ,86 schools in the district: 57 elementary, 13 middle or 
junior high schools, 9 senior highs and 7 special schools for handicapped 
students. To achieve racial balance, the school system operates a voluntary 
integration program featuring a wide variety of alternative curricula and an 
open enrollment policy. 

In 1982-83 the district had a general flind operating budget of $138 
million. This was supplemented with about $8 million of federal grant sup- 
port and $3 million of special state funding. 

The foundations of research and evaluation in Cincinnati, as in most 
other school' systems, are in the testing and survey movements of the 1920' s 
and 1930 ! s. Prior to this time, inquiry and . innovation rested almost 
entirely in the hands of the creative teacfher and administrator. In 1929, 
the Division of Research and Statistics was organized in Cincinnati Public 
Schools. Nine years later, a separate division in the Department of 
Instruction was set up to administer the group testing program. The$e two 
divisions provided the first semblance of systematic structure and scien- 
tific orientation for R&E in Cincinnati. 
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With the passage of the Elementary and Secondary Education Act, there 
came a new emphasis on program evaluation. The mandate that the federally 
funded projects developed under Title I be systematically evaluated led to 
the creation of the Division of Program Development in 1965. 

Since that time, research and evaluation activities have been conducted 
by personnel in a variety of organizational structures. For several years, 
the district had a Department of Research and Development headed by our cur- 
rent Superintendent, then an assistant superintendent. Jacobs was named 
Interim Superintendent of the Cincinnati Public Schools in July, 1976. 
Regular appointment as Superintendent of Schools followed in February, 
1977. 

As the district's financial condition worsened, some cutbacks in R&D 
~ as everywhere else — were essential. With much pain, Jacobs disbanded 
the department he had organized. Branches that had been part of the depart- 
ment were assigned to other departments. Functions were kept intact. 

Throughout the recent organizational changes, the Evaluation Branch has 
remained responsible for most R&E activities. Changes in the organizational 
structure have made it easier to coordinate some functions such as test 
administration and interpretation. But these same changes have also added 
difficulty and' chal lenge to the task of integrating evalution with program 
planning. ; 

The Evaluation Branch include s four sections: Program Evaluation, 
Testing, School Information, and Communications . These four sections coop- 
erate to serve district and school needs. Increasingly, however, a higher 
priority has been given to meeting the needs of the local school units in 
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the district. In' di scussing this process, I'll try to show how the four 
sections of the branch relate to one another and to the local school plan- 
ning effort. 

Early evaluations of Title I projects were organized to meet state and 
federal requirements. To receive continued funding, the school district had 
to complete state evaluation forms and submit written reports of results. 
Local interest in the findings thus submitted was limited, and evaluation 
outcomes did little to impact program development or administration. 

After several years, questions began to be asked nationally about the 
effectiveness of the Title I investment. This brought the local district 
increased pressure for improvement in these programs. Gradually, it became 
common to design evaluations so they served school district decision-making, 
as well as state and national reporting needs. Still, the perennial con- 
flict between scientific results and political expedience continued to limit 
the benefit derived from the evaluation effort. 

Somewhere around the beginning of^the 1970 ' s, our emphasis began to 
change. Individual schools in our district were seeking autonomy in plan- 
ning and implementing their instructional programs. The school district ob- 
tained a grant under Title III ESEA to create a "School -Community Evaluation 
and Development System." This system set up within one senior high school 
.area a working partnership between school a.id community to develop and eval- 
uate instructional programs. From this project there emerged a model for 
local school program development and evaluation which,- although never offi- 
cially adopted by the Board, has strongly influenced school district evalua- 
tion services. (2) 

T. Jacobs, James N. n A Model for Program Development and Evaluation." 
Theory Into Practice , February, 1974, 13, 15-21. 
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Local School Budgeting (LSB) 

LSB has been in Cincinnati for just over two years. Already we believe 
it has altered the traditional modes of educational planning. Although evi- 
dence' is still scanty, our current data suggest improved student achievement, 
in those schools where the program is working well. We also believe that 
these same schools are typically directing their educational resources more 
pointedly at their goals. If the evaluation effort can meet its challenge 
of adequately supporting this movement, the program holds great promise for 
getting more mileage from the educational dollar. 

To initiate the program, a Local School Budgeting Steering Committee 
was chosen by the community. This committee selected seven schools to par- 
ticipate in the first year. Three other schools were added to the program 
through a foundation grant. Additional schools are being included each year 
so that every school will be participating by 1985. 

Creative approaches to resource allocation is the basic challenge 
confronting LSB teams. The project has been designated a? a "budgeting" 
project to focus on this need. More accurately, it is a centralized 
process of educational planning that attempts to involve- the total school 
community of each local school. 

There are numerous examples of creative use of resources devised by the 
budgeting teams. Several schools have elected to organize volunteers to 
supply some of the school's maintenance needs, tnereby decreasing the jani- 
torial budget and providing more funds for instruction. Schools have formed 
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partnerships with local businesses and industry. They have organized pools 
of qualified volunteers to serve as substitute teachers so that money for- 
merly spent for substitutes 1 salaries could be dedicated to other purposes. 

One of the most common steps taken by school budget teams has been the 
application of energy-saving measures. Savings realized through such econo- 
mies are returned to the school for instructional use. Other schools have 
capitalized on the school district's current practice of providing pai -. for 
vol unteers- to redecorate classrooms and corridors when budgetary limitations 
preclude having sa4aried personnel perform this service. 

More related to evaluation activities are^be teams' efforts to 
strengthen the instructional process. Each team has an opportunity to 
locate personnel, material or training resources for thi s- purpose. When a 
local-site budgeting team attempts to stretch its resources to meet' the 
needs of students in the school, it is dealing with much more than dollars. 
The team needs to have some way to choose between alternative means of meet- 
ing needs; evaluation of possible programs is important. Techniques for 
program planning in times of austerity are also important to make the best 
use of limited resources. The team also needs to know of ways of soliciting 
funds from external sources, building volunteer support, and capitalizing on 
community and parent resourcesi Training and consultation on these and 
•other topics are available to all team members. 

To better serve the needs of students, the program locates decisions 
at the individual school. This decentralized process takes advantage of the 
day-to-day contact between teachers and students and uses that interchange 

19.3 
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to add valuable knowledge to decision making. Non-certif ica^ed staff mem- 
bers and members of the community also apply their experience and knowledge 
when assessl/ig the strengths and needs of their local schools. 

The rationale for LSB recognizes that parents are also prime teachers. 
They have specific concerns pertaining to thei r children 8 s^educati on. They 
are also uniquely aware of their children's needs. Since the public at 
Targe supplies the resources for education, they also have a legitimate 

•v 

interest in what is being done with those resources. 

Team members are selected by their respective groups: parents elect 
parents, community council members and staff choose their representatives. 
The principal, as. the accountabl e educational leader of the school , serves 
as a member of the team, but also acts as the„decision maker on many local 
issues. 

Team members bring ideas to the committee and reflect the position of 

v 

the group they are representing. This approach brings non-team members 

closer to the decision-making process. It enabl.es a parent v to contact a 

person on the committee and have his/her .opinion expressed. 

The underlying conceptual model for evaluating services .throughout the 

system is Stufflebeam's Context, 'input, Process, Product (CIPP) Model,. (3) 

This model sets forth four stages for evaluation as a means of "delineating, 

obtaining and providing information for decision making.". The first stage 

is context evaluation, which involves needs assessment and goal develop- 

» 

ment. Input evaluation, the second stage, includes the review of various 

3^ Stuff lebeam, D. L., et al . Educational Evaluation and Decision Making . 
Ithaca, Illinois: Peacock Publishers, Inc., 1971. 
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alternatives that might accomplish the goals that have been selected. In 
the third stage, process evaluation, the implementation of the selected pro- 
gram is monitored to insure that plans are carried out according to sche- 
dule. Finally, product evaluation looks at the outcomes of the educational . 

i 

program to determine whether objectives and goals have in fact been at- 

tained. ' . . 

The LSB planning model has been designed 'to fit the CIF? approach to 

Evaluation. It also includes four stages: determining need f . and goals, 
resource allocation, implementation, and review. 

V Key support for the local teams is provided by the Evaluation Branch; 
the Communications section coordinates .this support. This coordination in- 
cludes training and consultation. In addition, a parent-community coordi- 
nator helps principals and teams build local interest. Communications 
specialists garner media support and publish information to facilitate the 
work of the teams. ^ 

' • Other sections of the branch provide support services more tradition- 
ally considered evaluation .functions. These services are reflected in the 
discussion of the four stages of the planning model. 

Needs and Goals . Assessing need^ and setting' goal s are continuing 
processes." Formally, however, local needs assessment begins in January of 
each year. A goal development model which we evolved from a former federal 
project is recommended to each local team. The- School Information Section 
provides Vrucial service in this' stage of planning. A data bank called the 
School Information System includes .everal hundred variables on each school 
unit/ A number of these e.g., dropout rates by cause, socioeconomic 
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variables, etc. - r are very helpful in determining high-priority local 

needs. Also in the data tfenk are the results of annual surveys. Each year 

surveys of admtnistrators, teachersf parents, af& t students are conducted 

throughout the school system. Results of these surveys are useful 

barometers of attitudes and opinions. * ^ 

Resource Allocation . Once goals have been set, the team next addresses 

the allocation of resources. Monies allocated to the local lev-el are consi- 

dered first; the team determines the best use of these- dol lars in attaining 

r ■ 
the goals that have been set. Respurce allocation goes beyond the available 

dollars, however, as teams then address other ways to met their goals. 
Plans for obtaining external funds, using community resources, building vol- 
unteer resources, and teaching parents to work with their children toward 
school goals are all part of this stage. 

In addition to training, grantsmanship- assistance, and community rela- 
tions help, the Evaluation Branch provides support in prioritizing and eval- 
uating possible program strategies. A f ecf#al ly funcfed Cost Effectiveness 
project has generated a model by which teams' may study alternative program 
possibilities. A library research capability within the School Information 
Section furnishes guidance from the professional literature. 

The resource allocation stage of .the model is seen as the focal point 
for evaluation support. From the team's evaluation in this stage flow re- 
commendations and decisions that determine the 1 ocal program. 

Together with the implementation (next stage), these functions consti- 
tute the power cycle that lies at the heart of LSB success: Evaluation, 
Recommendation, Decision, Implementation. The LSB philosophy recognizes 

* 



\ power resides in shared participation in these functions. It minimizes 
uportance cfquestions liKe "Who makes the final decision? 1 * 
The final local school budget for the coming year is developed by the 
budgtu team, then presented to the public at open hearings at each school. 
A\ ter public feedback has been taken into consideration, the final budget is 
■nf - in to the central administration for approval. If the budget is% unac- 
ceptable, the administration discusses the problems with the budget team and 
together they resolve the issues. Finally, the Cincinnati Board of Educa- 
tion adopts the annual district budget reflecting the individual school 
needs. * • , • v> ) 

Implementation a nd Review . The two remaining stages of the planning 
— ' » ■ — * 

model are implementation and review. The evaluation support for these 
stages follows the traditional modes of process' and product evaluation. The 
Program Evaluation section, using data from the Testing and School Informa- 
tion sections, furnishes this support. A 

Implementation and review data are made available by several additional 
evaluation-related activities. The standardized testing program provides 
Survey achievement information in reading and mathematics, grades one 
through eight. The Program-Evaluation section has devised several creative 
ways to tie this information more directly to instruction. 

Other activities supporting implementation and review are a program of 
criterion-referenced instructional management called the Cincinnati 
Instructional Management System, an embryonic system of competency-based 
instruction, and a school improvement project using techniques developed by 
SRI, Incorporated. The Evaluation Branch works with other divisions of the 

y 
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school system, chiefly the Planning and Development Branch, in helping local 
schools use information from these activities." 

Training provided for team members covers three main topical areas: 
Educational Planning, Group Process, and Budget and Finance. The nine 
training modules offered in Educational Planning reflect the broad scope of 
the local school budgeting process: ■ 

- Needs Assessment 

- Setting Local School Goals 

- Prioritizing and Evaluating Possible Programs 

- Making the Best Use of Limited Resources 

- Soliciting External Funds 

- , Building Volunteer Support 

- Strengthening the Program through Community Resources 

- Capitalizing on Parent Interest 

- Evaluating Program Outcomes 

' Although those titles might suggest that we are trying to make expert 
program planners out of. all LSl$ members,* such is not our intent. Indeed, in 
some communities, we are satisfied if a quorum of a team membership attends 
most meetings. On the other hand, we do have some communities in which the 
teams have built on the training content with an enthusiastic and dedicated 
eYfort to tailor the program to the needs of their students. 

Wh^t cha^acterizes N th^ settings in which LSB works best? We have found 
that the single most important component of a successful LSB effort is the 
wholehearted commitment of the local school administrator. The principal is 
most important because he/she alone^an weld the staff, community, and 
school clientele into a cooperative unit. The process sinrly does not work 
in 'settings where principals give only condescending endorsement to collabo- 
rative planning. 

A second obviously important ingredient oC LSB successs is community 
interest. Our underlying assumption has been that ndne of our local school 
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communities is without an adequate degree of interest in education to permit 
the process to work. In a few instances we have yet to validate this 
assumption. We have reason to believe, however, that distrust on part of 
some segments of the community is*a major reason for non-participation. 
Also, many residents of the poorer sections of the community seem to doubt ^ 
their own ability to contribute to such a process. 

A question we have often been asked is, "What does a process like LSB 
est? That's hard to answer. In one sense, it costs everything because the 
scope of the planning effort eventually entails every element of the dis- 
■trict's bidget. In another sense, it costs J nothi ng. The entire process has 
been effected by realigning resources and setting new priorities. We have 
spent no more money on LSB than we would have spent without it; remember, • 
our poverty. was a key motivator for taking this route in the first placey 
We believe that the effort to better direct our educational resources to the 
needs of students at the local school level can only increase our cost- 
effectiveness. 
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The )Local School Evaluator 

° The agent serving the evaluation needs of the schools is called the 
Local School Evaluator (LSE). This .is a central ly-based staff member 
assigned responsibility for helping the local school evaluate its educa- 
tional program. LSE ; s have been functioning in Cincinnati for ten years. 

The position of LSE was first operational i zed within Title I. Approxi- 
mately half the 'resources formerly invested in the centralized evaluation of 

199 \ 
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Title I were diverted to serve local school needs through this position. 
Lack of available funds has limited the implementation of this function in 
ather school settings. With the advent of LSB, however, the service has 
been extended. 

^Each LSE is assigned a group of schools whose respective evaluation 
needs he/she must serve. Early in the v school year, the LSE makes contact 

B f 

with the assigned schools and outlines available services. The functioning 
of the LSE for the year is determined in large measure by the specific ser- 
vices requested by the local school. 

In cases where response to the initial contact is slow, the LSE will 
attempt to stimulate further interest by offering assistance in addressing 
specific local school problems. These offers frequently take the form of 
letting one school know of a service that has proven valuable in another 
location. The history of local school evaluation in Cincinnati indicates 
that the' services of the LSE are valued as a means of improving instruc- 
tion. 

Ahn, Barta, and Rockwood have described the LSE function as an evalua- 
tion track of services supporting the instructional track. Needs assessment 
supports the selection of content and goals; diagnosis of entry behavior 
supports the specifications of objectives. Assessment of implementation 
supports the selection and implementation of strategy, group, time, space 
and resources; and assessment of end results supports the instruction pro- 
cess itself. (5) 



"5^ Ahn, Unhai R. , Barta, Maryann B., & Rockwood, Stacy "Localizing 
Evaluation for Public Schools: Concept and Practice." Presented at Annual 
Meeting of the American' Educational Research Association, San Francisco, CA 
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To give the reader a better understanding of how the LSE assists the 
local school's planning effort, let me describe six specific services selec- 
ted as representative of LSE functioning. Three of these serve the process 
and product of evaluation functions and three the context and input evalua- 
tion functions. 

The examples will reflect how the LSE interacts with school staff, in 
addition to the LSB teams. Staff interaction is an important part of the 
LSE function; its significance may be obscured by the emphasis of our 
earlier discussion. 

Alternative Data . Special procedures for evaluating Cincinnati's al- 
ternative programs call for an evaluation team to study each program on a 
cyclic basis. The LSE is required to furnish the evaluation team with rele- 
vant data. In the course of the team visit, the major focus is on process . 
and product evaluation. The visit begins with a discussion of data that the 

y 

LSE has assembled, and the interpretation of this information by the evalu- 
ator is crucial to setting the tone for the visit. 

Data forms have been devised for summarizing the data on the alterna-. 
tive programs in accordance with the set of ^lternative questions prescribed 
for team responses. Of primary interest here are those questions that gen- 
erate comments and ratings to be shared only at the local school level. 
These questions concern curriculum, administration staffing needs, space 
utilization and concerns- of persons involved. 

For example, theJ.SE provides data to answer the question, "Does the 
status of reading achievement scores suggest? a change in overall program 
emphasis?" Another question related to curriculum is, "Do item data reflect 

2M ' 
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certain skills that need increased emphasis or modified approach?" Related 
to student concerns, the evaluator compiles data to answer the question, "Do 
students 1 survey results suggest specific concerns that the program-or staff 

s 

should address?" In the course of the visit, team members also seek an 
answer to the question, "Does casual student questioning reflect other 
matters of important concern to students?" 

CIMS Reports , Another service of the LSE related to process and pro- 
duct evaluation is assisting staff in interpreting and applying reports from 
the Cincinnati Instructional Management System (CIMS), This is a computer- 
ized system of monitoring student progress in achieving fundamental reading 
and arithmetic skills. The teacher using ClftS inventories can more easily 

4 

keep track of each student's progress in skill mastery. The evaluator can 
assist the teacher in understanding the computer-generated reports, recog- 
nizing common needs of students and planning strategies to serve these 
needs. " ' 

One CIMS report, for example, i ndicates* students' mastery levels on 
each of a variety of reading skills. These include skills in phonetic ana- 
lysis, structural analysis, vocabulary, literal comprehension and interpre- 
tive comprehension. If several students .reading at approximately tjie same 
level show a weakness in distinguishing hard and soft cj_ sounds, the teacher 

can arrange practice in that skill for these students. It is this kind of 

f 

focus that the evaluator helps to make possible in working with CIMS. 

Chapter I School Profile . Because of the availability of resources for 
Chapter I evaluation, each Chapter I school has a larger amount of data than 
• other schools in the system. Over the last several years, LSE's have worked 

/ 
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to help school staffs use this information for instructional improvement* 
One important part of this approach has been the Chapter I school profile. 

Essentially, this profile helps the staff of an^indi vidual school eval- 
uate certain process and product outcomes in the light of comparison with 
other Chapter I schools. This has been a recommended approach among these 
inner-city school units, which formerly suffered from comparison with 
schools in more affluent communities. 

The school profile includes information on student achievement as mea- 
sured by. standardized tests. Information on student attendance; teacher, 
parent, and student attitudes; data on parent involvement — all of these 

are included in profiling the Chapter I school against the background of 

• * 

other schools in the program. 

Item Report . One of the most frequently requested services of the LSE 
is training in the use of standardized test item analysis. The LSE helps 
teachers use the results of specific items to diagnose student and group 
needs for instruction or remediation. , . 

Several approaches have been devised to make this process easier for 
teachers. Among t^hese are the pupil item report worksheet \ and the group 
report worksheet. The pupil- worksheet allows the teacher to list names of 
students whose item results suggest a need for more practice time, further 
diagnosis or other reinforcement. 'The group worksheet permits the compari- 
son of class unit or school unit item data with similar data for tlae schodl 
system. 

Survey Information . Our earlier discussion of goal setting mentioned 
the work of the LSE in helping local teams use information collected through 
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the annual surveys. The surveys of administrators, teachers, parents, and 
students provide a large volume of attitudinal information which the LSE can 
help the local team understand and apply. 

The student survey, for example contains a number of items related to 
self-concept. By comparing the results of these items from one year to the 
next and- with results from other schools, the team can gain useful insight 
into specific needs for program emphasis. 

S chool Information System . The survey results on each school are 
stored in the district's School Information System. This data bank includes 
several hundred variables on each school unit. A number of these, e.g., 
drop-out rates by cause, socioeconomic variables, etc., are very useful in 
context and "input evaluation. 

Each year the school receives a comprehensive report of its data as 
contained in the School Information System. The shesr volume of this report 
makes it important that someone be available on a consulting basis to help 
staff and team interpret and apply the information. 

A Look Ahead' 

What we have seen happen in our attempt to support community efforts to 
improve instruction has been encouraging. We see considerable basis for 

hope in this approach. As the educational dollar continues to become more 

j 

scarce, the communities to whom .the schools belong are joining in the 
concurrent missions of providing adequate resources and putting what we have 
to the best possible use. 
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c ' Advisory committees are not new, of course. Many, districts Have used 
them, some under requirement of state law, and they have had varying degrees 
of success in realizing educational benefit from the wo^c of these groups. 
We see. our emphasis and procedure as somewhat different from those of most 
other districts. And the support evaluation provides to these teams is a 
key element in this difference; 

Perhaps more than in any other district, communication of evaluative 
data to local clientele is given top priority in Cincinnati. This emphasis 
has led to'long-range planning in which research, testing, and evaluation 
functions are the means of obtaining information to be communicated to those 

interested in the schools. 

The five-year plan for the school system includes a major section on 
communications. Four goals are set down for this part of the plan: 1) im- 
prove the school system's ability to hear the voice "of the pub Vic it serves; 
2) obtain better information to serve the need of staff, clients and commu- 
nity; 3) improve the quality of 'information processing and reporting; and 
4) assist staff, clients, and community in interpreting- and using informa- 
tion for decision making. 

Our attempt to support community improvement of instruction has not 
been free of probjems. Stirring up' apathetic communities, gaining commit- 
ment 'of sufficient time, internal coordination — these and other difficul- 
ties have kept Vis aware that we are still in a Veal world where panaceas 
seldom exist. We deal with these problems as they appear, using the re- 
sources described in this exposition. The success we have had leads us to 
believe that some of what we are doing may be helpful to managers in other 
districts., « 205 
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A superintendent who shares Cincinnati's pressing financial dilemma may 
be able to apply' our strategies for gaining additional community. support . 
Another manager confronted with non-producti v£ advisory committees could 1 
perhaps use some of Cincinnati's techniques for involving them more inte- 
grally in educational planning. m „. 

Whatever the application, I hope my attempt to describe the LSB pro- • 
cess, giving the answers to many of the questions we are asked about it, 
will be of some utility. I hope that the reader can benefit from what we 
have learned about the key role of the local school administrator in apply- 
ing such a process. I hope*that our experiences in trust-building, working 
toward consensus, . and applying evaluative findings to program choices will 
be helpful . 

I hope, too, that the reader can profit from some of the mistakes we 
have made. It js important to avoid the pitfall of trying to^accomplish too 
much change too rapidly. Especially with a process such as this, it is cru- 
cial to allow time enough for the participants to work through many "details 
of operation. Involving all parties, affected is e -constant challenge; And, 
most importantly, the selfish interests and protected domains of individuals 

.must never be allowed to interfere with what is really best for the students 

\ \ 

we serve. V 
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CHAPTER 5 



School District- Analysps 

in which the impediments .to developing such 
systems are discussed along with the conditions 
propelling districts in this direction . • . 
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CHAPTER 5 
SCHOOL DISTRICT ANALYSES 

Introduction 



In this chapter we take a broad conceptual view of why school districts 
do not naturally link their testing and evaluation with their instructional 
programs, even though this may seem a logical or important Visage to deve- 
lop* We include this section for several reasons: one, we believe that you 
cannot improve or change something unless you understand it; two, we want to 
dispel some common misconceptions about how school districts operate and 
change; three, we want to stress the importance of school o:stricts develop- 
ing their own approach to lfnking testing, evaluation and i nst ruction Tcrther 
than adopting ready-made programs. 

First, let us consider the steps commonly taken by school districts 
tKat are. found with low or declining test scores. Again, we will refer back 
to the scenario in doing so. 

Why Districts Have Trouble Dealing With Test Scores v 

> 

In developing this section, we will start off by referring back to the 
scenario in Chapter 1. While the scenario focuses on the decline of test 
scores, it also contains several elements that are typically found in school 
districts and that complicate attempts to reach a solution to this test 
score dilemma. For example, consider the following: 

1. Lack of consensus over the problem's cause. A district's 
declining test scores can be caused by one\>r several fac- 
tors: poor instruction, inadequate time on instruction, ** 

\ 
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limited match between what is taughfand what the test 
measures, statistical^ variations such as standard error, 
lack of pupil knowledge of how to take tests, shifting 
pupil population. Each of these variable causes suggests a 
different .solution or strategy. 

These declining test scores are not the only problem the 
district face^ The electrical power failure in high 
school demands immediate attention. The district is expe- 
riencing declining enrollments which will likely force the 
district to initiate the difficult task of closing schools 
and possibly ^laying off staff. Both of these processes, 
closing schools and laying off staff, are generally politi- 
cally and organizationally difficult problems that will 
likely consume much time and effort. Of course, districts 
often face other problems, e.g., desegregation orders, drug 
problems, contract negotiations, that also chip away at the 
amount of time and energy that testing problems can 
receive. 

The district does not have the luxury, or freedom, to pre- 
sent the problem to the public in its own terms. In this - 
case, the newspapers presented the story and likely influ- 
enced considerably the way the the public interpreted the 
results. Newspaper reporters generally are not educators 

209 



- 205 - 



and sometimes, but by no means always, their interpretation 
of educational events can be limited or reductionist in 
character, e.g.,- the teachers must not be teaching the 
basic skills. While lack of teacher attention to basic 
skills may not really be the' problem, the district nwst, 
nonetheless, deal with the public's perception that this is 
indeed the problem. 

School management cannot take unilateral steps to solve the 
problem. In this scenario, the school board, or a repre- 
sentative public body, and the teachers' union, which re- 
presents, the school teachers, will likely have a major say 
in whatever is decided. The board, for example, if it is 
concerned about re-election, may insist that the district's 
solution to the problem be in response to the public's per- 
ception of the problem. Thus, they may insist on more 
classroom time on basic skills. The teachers' union, on 
the other hand, may insist that teachers be paid for any 
ou^-of -school inservice training sessions. If the district 

is facing budgetary problems, this may effectively remove 

■■i ■ , t 

inservice training as a viable strategy for solving the 
problem. 

Lack of consensus on the solution to the problem. If* one 
can not define the cause of the immediate problem it 
follows that it will be impossible to select an Appropriate 
solution to the problem. But even if one can reasonably 
define the problem, there is limited consensus about* what 
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to do about it. For example, if the problem is defined as * 
inadequate t'ime on instruction in basic skills, does that 
mean that all pupils should increase their time in basi<jf. 
skills instruction or just a few? The instructional time 
in schools sis finite; if instruction in basic skills is 
increased, what other- subjects will "lose" instructional 
time? How will that dilemma be resolved? 
Given all these characteristics of school districts, is there nothing 

* • > 

hat can be done? Must school distfncts - continual ly .be buffeted back and 
orth on the basis of test scores? Must they forever be placed in a dilemma 
here they cannot define the problem, have limited choice over solutions, 
nd have no realistic way of determining whether or not selected solutions 
re achieving the intended results? 

Some districts, in response todeclining test scores, take specific 
iteps, such as: teaching the pupils how better to take tests, or increasing 
;he required classroom time on certain subjects such as reading or math, or 
initiating yet another series of inservice training workshops- for teachers. 

**** *■ 

Je would argue, these isolated steps generally do not make much of a dif- 
r erence. These steps do not usual ly" improve test^ scores very much and the 
iilemmas, and sometimes risks, remain. In the mean time, teachers begin to 
lose their feeling of efficacy, the public lessens its support for the 
schools and the pupils are judged as not receiving an adequate education. 

We think not. While the problem is complex, it must be solved if dis- 
tricts are going to make progress in providing a good educational program. g 
r/hile at the same time improving public confidence and satisfying staff con- 
cerns. But it is not easy. School districts, faced with such a problem as 
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jthat described in the first chapter's scenario, are confronted with a 
formidable problem. As the superintendent and board members and staff 
* members in our scenario meet to consider the problem, they must address 

V ? ^ 

several quests ons . 

How can N we describe the problem? Where do we start? Who should be 
involved? How wfll we know if we are making progress or have solved the 
problem? The remainder of this chapter deals with some basic assumptions 
that we feel should be kept in mind when this or any school district 
approaches this problem. 

Some Assumptions 

As the administration, board members, and staff members iJ our scenario 
school district sit down to begin to ponder their dilemma, we would recom- 
mend that they keep in mind some assumptions that should form the basis for 
their discussions. We will list the assumptions here, and then subsequently 
discuss each in more detail immediately thereafter. 

Assumption 1. The problem cannot be solved unless it is first 
analyzed and understood. 

Assumption 2. The problem is a school district systems problem 
and not the fault of any one individual, groups of individuals 
or single process. 

Assumption 3. A strategy for linking together the district's 
testing and evaluation and instructional program can be an 
effective strategy for solving the challenge posed by- declining 
test scores. 

Assumption 4. A system for linking testing and evaluation and 
instruction must be situation-specific and accommodate various 
external and internal factors. 

Assumption 5. Developing and implementing a subsystem for 
linking together the district's testing, evaluation and in- 
structional programs (t/e/i linking subsystem) can be thought 
^ of as an educational innovation. 
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Assumption 1, The problem cannot be solved unless it is first analyzed 
and understood . 

This seems like a self-evident statement; ye\ we have observed and 
studied district after district that has adopted a particular strategy or 
taken certain steps without hav.ing taken this first problem-identification 
step. In 'order to satisfy public demands for action districts often choose 
a single strategy such as mandating that instructional time on^selected sub- 
jects be implemented, or changing the district's reading or math or social 
studies program, or requiring teachers to attend mandatory inservice pro- 
grams, or changing the norm-referenced tests the district is using. This 
list of selected isolated activities could go on and on. The district's 
basis for these seemingly isolated strategies might include such factors as 
an influential book salesman or a local university consultant or spme unex- 
amined belief system of a prominent school board member or administrator. 

In making these statements we don't mean to be hyper-critical of school 
districts who act in this manner. As we pointed out earlier, and as was 
illustrated in the scenario, schools districts have many problems pressing 
on them for solution. Often there simply isn't time to give this problem 
the sustained attention it needs, especially if the district is unsure of, 
or lacks the technical capacity to identify and deal with this problem. 
Yet, the problem will not go away and its potential ramifications can be 
serious indeed. 

Let's look again at the scenario district for ways in which the 
district might begin looking at the problem. The most immediate question 
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is, "What is the problem?" § or perhaps more accurately, "What are the prob- 
lems?" Many districts often assume that the lowering test scares automati- 
cally mean that the students have learnedi less or that the quality of in- 
struction has declined. Of course, it may not be an instructional or learn- 
ing problem at all. It may be that there is'a whole relationship between 
what the test measures and what is taught in the classrooms. Either the 
instruction should change 9r a different test might be selected. Or dis- 
tricts make comparisons between test scores from one year to years past. If 
there is a decline, they assume that the schools are not performing as well 
as in years past. But is the school population the same? .For example, 
there may have been a significant influx of new students whose previous edu- 
cational background is substantially different from the district students 
who were tested earlier. What the test results might indicate is that the 
district test does not measure what new, or relatively new students learned 
elsewhere. Or, a district might conclude that the instructional program 
simply doesn't give adequate time to certain subjects. But often times, 
districts have nb basis for knowing how much time is presently being spent 
in classrooms on various subjects. Or a district might conclude that a pre- 
sent instructional program isn't effective, but they have little knowledge 
about whether or not the program has been implemented by the teachers. as it 
was designed. « 

In a sense, what starts out to be an essentially straightfoward problem 
quickly dissolves when various alternative problem statements are con- 
sidered. In a sense, the "testing problem" becomes an evaluation problem. 
The situation crtes out for a thorough systematic assessment. The 
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dilemma will riot likely be dealt with effectively until such an analytical 
process is begun. Please note, we are not suggesting that such an analysis 
is a single, massive effort to analyze the problem. 1 Rather we are suggest- 
ing an analytical frame of mind that will result in long-term, continuing 
inquiry into the problem, and the results of various strategical implementa- 
tions to solve the problem. 

Assumption 2. The problem is a school district systems dilemma, and 
not the fault of any one individual, groups of individuals, or single 

person . - t 

School districts, even small ones, are complex organizations. It is 
difficult, often, to conceptualize the problem because we limit our thinking 
to the district organization itself, with its school board, school sites, 
and administrative and teaching staffs. Often, we focus our attention on 
some subsystem, such as the Central administration or school sites when con- 
ceptualizing the problem, ^ut, of course, school districts do'not exist in 

J 

isolation from their environments. Nor do the various subsystems exist 
independently of each other. 

But how does one conceptualize such a" complex entity? We find the open 
systems model of organizations to be useful ->in this regard. 

School Districts As Open Systems . Open systems is a widely-used and 
relatively simple conceptual model. It is used in many fields, such as 
biology, to illustrate the functioning of an organism in its environment. 
One of its major features is that it accounts for the external environment's 
influence on the unit, or organization being studied. By using the open 
systems model, one is forced to consider the dynamic interaction between an 
organization and its environment. 
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Viewed graphically, a school district as an open system can be depicted 
as follows: 
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Relevant External Environment . This includes those agencies, indivi- 
duals, groups of individuals and other organizations that can influence an 
organization's functioning. In the Chapter 1 scenario, this was represented 
in several ways: parents calling the schools, the newspaper article, commu- 
nity persons. Of course, the list of external environmental influences is 
much larger, and includes such groups and individuals as: the federal and. 
state courts; federal, state, and local governments; taxpayer associations; 
radio and television; local community and business leader's; post-secondary 
schools to which the district's population matriculate and local private 
schools. From time to time, these groups -and individuals can exert consi- 
derable pressure on the school district to confrom to its or their wishes. 
Many factors influence the district's response including the external 
groups, political power, legal mandates, and the saliency or popularity of 
the issue. What is mpre, the district's response to these demands, and, 
some have argued, the district's internal organizational structure, must 
respond to the make-up and complexity of the external environment if the 
v organization is going to deal effectively with its problems. 

Permeable Boundary . All organizations have a boundary around them that 
separates the organization from its 3 external environment. It is, however, 
almost impossible to determine exactly where that boundary is. In the case 
of the schools, for example, the governing body of the district (that is, 
school boards) are agents of the state. Does it include the parents who in 
many i /stances-have final authority -over the child's education, but limited 
authority in other instances? This issue is essentially unresol vable, 
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depending as it does on the problem or action being considered and how one 
defines influence* 

This boundary is permeable, that is, virtually no organization is com- 
pletely .independent of its external environment. External influences^ can 
penetrate the boundary and effect its operations and decisions. But the 
permeability of the boundary 1 " varies considerably from organization to orga- 
, nization. In the case of public school districts, the boundary is quite 
easily bridged, That is, many of the school's actions on such vital matters 
as budget allocations, whether or not to close schools, the school's test 
scores, are easily accessible to the external environment. What is more, 
decisions by external agencies such as federal courts or state legislatures 
and local business, for example, that may decide to open a new factory, 
often bridge the school district's permeable boundary and require that the 
organization vary its processes or decisions. 

In our scenario, this boundary penetration is clearly illustrated by 
the way in which the test score data were obtained by the press and subse- 
quently by the effect the newspaper article had on the external environment 
and its suSsequently penetrating the organization to cause it to consider 
the appropriate action to be -taken. This boundary permeability has both 
advantages and disadvantages. On the plus side, it forces an organization 
to be responsive to items and issues that the external environment considers 
important. Organizations cannot ignore the external environment and hope to 
remain current with the needs of the constituents they serve. On the nega- 



tive side, the organization can be easily buffeted by changing public atti- 
tudes that may be based on incomplete information or on faulty analyses, or 
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on developments elsewhere that may be inappropriate to the school district's 
situation., The result of this is that school districts find it difficult to 
conduct long-term planning; a plan Lhat is developed and implemented based ^ 
on the public's view or wishes at a given time may have to be altered or 
discarded'before the plan is fully implemented because some influential 
agency or group in the relevant external environment lias changed its goals 
or priorities. .« 

< 

Internal Subsystems . All complex organizations, such as school dis- ^ 
tracts, are composed of subsystems. This factor is sometimes forgotten when 
we^think and speak of organizational actions and decisions. Then, we say a 
school district did this or that, or performs in certain ways. We would be 
more accurate to say that various subsystems and individuals have decided 
something or are performing <n a certain way. Thus, a school district can 
have a curriculum that is adapted by the school board (one subsystem), arti-. 
culated by the curriculum department (another subsystem), and implemented by 
the teachers in the school sites (another subsystem). And, as .will be ela- 
borated in a 'subsequent section, the articulation and coordination among 
these various subsystems is often less than perfect. Thus, one can talk 
about the district's curriculum, but general statements about it often 
ignore or gloss over the complexity that is inherent in such a general 
statement. 

A complex organization then consists, internally, of a whole series of 
subsystems that vary considerably in the degree to which they are working 
toward common goals and the level of coordination and cooperation among 
them. / \ 
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Input, Throughput and Output * This organization of operating subsys- 
tems, which exists in a dynamic relationship with its relevant environment, 
then, takes in resources (inputs in "the form of money, materials, personnel, 
clients, etc. which it parcels out into organizational subsystems, in the 
form of such agencies as school sites, personnel offices, curriculum unit's, 
budget offices, custodial and support staff, etc., to produce an output, 
such as institutional programs, educated pupils, employed citizens (both 
graduates and citizens who work in the district), community entertainment 
(such as athletes and musical performers), and babysitting (that is, pupils 
are engaged and supervised during the school day). 

While there are commonalities from district to district on these in- 
puts, throughputs and outputs, significant differences exist among districts 
that make generalizations hazardous. That is, schools differ considerably 
on such important items as: level and source of financial support, mix and 
variety of pupils in racial and socioeconomic terms, district organization 
of the subsystem, coordination among the subsystem, definition of a success- 
ful graduate and organizational goals. 

, Summary . We believe this open systems model of school districts pro- 
vides a comprehensive way to look at how a school district functions and the 
various initial factors that must be considered if the organization is to 
deal effectively with the problems and challenges it faces. In the case of 
our Chapter 1 scenario, one begins to see the interplay among the various 
'parts of an open system. The district is seen as a complex of internal 
subsystems .hat are being effected dramatically by the dynamic interchange 
of the organization with its relevant environment across its permeable 
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boundary. The various subunits are being called together to forge a 
response to the challenge — the declining test scores. 

One final comment. It is important to note- that this is an open sys- 
tems model , and not a theory . That is, it provides a way to conceptualize 
how complex organizations function in their environment. It provides, we 
believe, an effective way to view the organization's functioning but it does 
not in any way pre>i^t4iowai)/organization does or should respond to exter- 
nal challenges. Organizations indeed respond in many ways to their relevant 
environment. The exact relationship between their response and their orga- 
nizational success is at best speculative. We believe that one way the dis- 
trict can respond to the declining test score challenge is to consider link- 
ing its testing and evaluation -program to its instructional program. It is 
to this process that we turn next. 

Assumption 3. A strategy for linking together the district's testing 
and evaluation and instructional program can be an effective strategy for 
solving the challenge posed by declining test scores . 

The wording of this assumption is important. Note that we have said' 
that a t/e/i linkage strategy can be an effective strategy for solving the 
challenge posed by declining test scores. 

We are not suggesting that this t/e/i linkage is the only way a. dis- 
trict can meet this challenge. Some districts might, for example, simply 
not give norm-referenced tests and release the results to the public. 
Others may decide lhat strategies that could be considered "humanistic" 
would be more appropriate. We have no arguments with those who would_select 
such approaches. Our advocacy of this t/e/i linkage strategy is based on - 
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wh^t we perceive as the public's belief in the importance of norm-referenced 
tests, or criterion-referenced tests, or competency tests or some combina- 
tion of these, (For a more detailed discussion of tests and what they mea- 
sure, see pp. . ) For good or bad, testing has become accepted practice 

in American schooling. Given the pervasiveness of this practice, it makes 
sense, we feel, for districts to maximize the potential usefulness of those 
tests in informing decisions about the instructional program. 

Program Evaluation, though a more recent phenomenon, is also a common 
practice in many school districts. Largely in response to federal and state 
evaluation mandates that accompanied external funds for special programs 
(such as ESEA title I, ESEA Title IV and PL 94-142), school districts have 
either developed a capacity for in-di strict program evaluation research or 
they have developed a continuing relationship with an external evaluation 
agency that can provide systematic evaluations. Many districts now have the 
capacity to perform evaluations and, with some modifications, these evalua- 
tions can serve to inform local school district instructional decision 
making. 

Also note in our statement of this third assumption that we refer to 
this as an "effective strategy for solving the challenge posed by declining 
test scores." This wording intentionally refers to the declining test 
scores as a challenge. We do not take the position that the declining test 
scores are themselves the problem. Instead we view them as symbols or mani- 
festations of the instructional program. Our interest is in the information 
presented by test scores and through evaluation to improve instruction. A 
district could have an excellent instructional program and poor test 
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scores. Similarly, a school 'di strict could improve its test scores, through 
various manipulations of students, teachers and statistics, and make no in- 
structional improvement. The key concept here is that the testing and eval- 
uation activities and capacity can be linked together to inform instruc- 
tional decision making, and thereby improve instruction. 

Our basis for suggesting this t/e/i linking strategy is based on devel- 
opments on a national scale and at the local level in many districts in 
testing and evaluation. 

In testing we have made considerable progress in improving the quality 
of norm-referenced tests and in understanding their strengths and weaknesses 
in providing information about instructional programs. In short, there is 
no need for districts to blindly react to norm-referenced test scores as 
though they are the ultimate^measure of a school district's program. The 
fact that some districts or communities continue to use norm-referenced test 
results in that way suggests that the present sophistication regarding those 
tests' strengths and weaknesses is not uniformly known or adhered to. 
Norm-referenced tests can provide important data but only if the relation- 
ship'between the norm-referenced tests' content and the school district's 
instructional program is known and understood. 

A parallel testing movement has developed in the design and use of 
criterion-referenced tests. These tests can be more specifically related to 
a district's instructional program and accordingly they can provide almost 
immediate feedback regarding the district's program. But their potential 
use for informing instruction is limited to the degree to which they are 
linked to the instructional program. As with norm-referenced tests, there 
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are certain strengths and weaknesses associated with criterion-referenced 
tests. But if properly designed and linked to the Jnstpuctional program, 
they can provide valuable data. 

/ Still a third moyement ? is the competency testing movement. While some- 

A 

what more limited in scope than the norm-referenced and criterion-referenced 
tests, these too can provide valuable information about the instructional 
program's ability to provide a basic survival education. Data collected 
from these tests can answer important questions about the school district's 
instructional program and the students' progress in achieving basic survival 
skills. No one will argue that this proficiency testing program should be 
the ultimate goal for the district, but it does provide a base line upon 
which the district can build. And, data from these tests can provide valu- 
able information for district instructional decision making. 

We make no argument for one testing system over another. We only re- 
mind the readjer that each test has its strengths and' weaknesses; accord- 
ingly, each provides its own kind of data (see pp. ). And these data can 

form an instructional management information system but only if they are 
adequately linked to the district's instructional program* 

Program Evaluation is a more recent development in education. The main 
emphasis for its development came from requirements that accompanied fed- 
erally, and subsequently state funded programs for systematic evaluation of 
such projects. The purpose of these evaluations was to provide federal and 
state officials data upon which to make decisions about whether or not to 
continue program funding. 
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These requirements started a sizable movement in education. A number 
of scholars, many of them trained in educational psychology and psychome- 
trics, began developing methodologies and procedures for program evalua- 
tion. . Educational evaluation began to develop as an identifiable field of 
study replete with a Center for the Study of Evaluation, its own division in 
the American Educational Research Association, graduate programs in major 
universities, and both research and pra'ctitioner journals. 

Paralleling, and indeed a major stimulant for this development of edu- 
cational evaluation as a field, was the development of school districts' 
capacities to conduct educational evaluations. In the beginning of the 
movement many school districts handled all the evaluation requirements by 
utilizing external consultants or by adding this tasl; on to the duties of 

those who handled the district's testing program. This procedure continues 

< 

in many districts, especially smaller districts, today. Many other dis- 
tricts began to consolidate all the evaluation funds into a separate unit 
which was given a variety ^of names which included some mix of terms such as 
research, evaluation and testing. 

The result of this movement was to provide districts with increasingly 
sophisticated capacity to conduct program evaluations. But recent research 
has revealed that much of the work of these units, that is, their evaluation 
reports, has been directed, understandably, to external funding agencies. 
The evaluation reports have been designed to answer the external funding 
agencies' questions ; they have not been developed to meet the local needs. 
Also, the evaluation^community has come to realize that in many instances 
the evaluations were not being utilized as intended by decision-makers. 
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There were many reasons for this, such as a misunderstanding of evaluation's 
role in decision making and the evaluators' adherence to standard research 
formats rather than developing evaluations that were responsive to the deci- 
sion makers' needs or pressing questions. In recent years, the evaluation 
community has gone through reappraisal of its purposes and methods in an 
attempt to increase the utility of evaluations for decision makers, Consi- , 
derable progress has been made in that regard. 

Thus, presently, many school districts have developed program evalua- 
tion capacities consisting either of groups of evaluation consultants or an 
internal research, evaluation and testing unit. And, this evaluation capa- 
city can provide much needed data on the school district's instructional*^ 
program if it is appropriately linked to it. That is, this capacity has 
habitually been used to ratify external funding agencies the challenge, 
and promise of school districts now is to forge a linkage between their 
evaluation procedures and local school district instructional decision 
needs. 

In summary, school districts are under considerable pressure to improve 
their instructional programs. Yet, in many perhaps most — districts, 
instructional decision making is done quite unsystematically, often in re- 
sponse to some set of scores from an annual administration of some set of 
norm-referenced tests. Yet, these tests have limited capacity for making 
well-informed instructional program decisions. This is especially true if 
the relationship between the norm-referenced^test and the district's in- 
structional program is unkown. What is more, most school districts, given 
their organizational pattern with teachers functioning quite autonomously 
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behind classroom doors, have surprisingly little data available about- what 
their instructional program really is and how it is implemented. But devel- 
opments in testing, such as refinements of norm-referenced tests, the design 
of criterion-referenced tests and competency tests, can provide districts 
with useful tools with which to assess the outcome of their instructional 
program. Similarly, advances' in the evaluation field make it possible for 
districts to begin to obtain systematic and comprehensive analyses, both » 
formative and summative, of the ongoing districts' instructional program. 
Together, this testing and evaluation capacity, if linked with the instruc- 
tional program, can provide a district with an instructional managment 
information system. The fact that this has not occurred in many districts 
is probably due to several factors including characteristics of school dis- 
tricts as complex organizations; a topic to which we turn next. 
9 Assumption 4. A system for' linking testing, evaluation and instruction 
must be situation-specific and accommodate various external an d internal 
variables . 

We' suspect, and indeed our research confirms", that few districts have 
linked their testing and evaluation activities with their instructional pro- 
grams. There are probably a number of reasons for this, such as not reali- 
zing the potential or the district's capacity to develop such a linkage or 
simply the district's having decided on another strategy for instructional 
improvement. We suspect, however, that the characteristics of school dis- 
tricts in general are such that they make the linkage of these three pro- 
cesses, testing, evaluation and instruction, difficult to accomplish. We 
suggest that the following four characteristics of school districts mitigate 
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against school districts moving naturally or easily toward developing this 
t/e/i linkage: institutionalization, goal differences, permeable bounda- 
ries, and loose coupling. 

Institutionalization . This term, derived from the sociology of organi- 
zations, suggests that some organizations are valued by the public as insti- 
tutions and given special status and recognition. Generally they receive 
public funding and a virtual monopoly for their activities. They do not 
have to compete, in the typical sense, for survival and they are under less 
pressure than most other organizations to justify their existence. Examples 
of such organizations would be national parks, public libraries and public 
schools. This special status, of course, confers to these organisations 
certain advantages in that they do not have to continually fight for survi- 
val although, to be sure, the level of their support is subject to political 
trends. They are by no means trouble-free. 

But this institutionalized characteristic has its disadvantages as 
well. That is, the organization can survive without looking critically at 
the way that it operates and the degree to which it is successful - in accom- 
plishing its goals. A" common outcome of this institutionalized character- 
istic is that institutionalized organizations often deal in symbols rather 
than in reality. Schools and school districts, when asked if they are of 
high quality, will usually respond by referring to such factors as: the 
fact that all the teachers are credentialled, or that an admi rably high per- 
cent of the teachers have Masters degree, or that the^Hbraries have so many 
volumes, or that the curriculum is clearly described. Seldom are questions 
raised as to whether or not credentialled teachers or teachers with higher 
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degrees are more competent than those without such symbols, nor do they - N 
really see if pupils are reading and understanding all those books or if the 
curriculum is actually being taught in classrooms. In other words, schools, 
as institutionalized organizations, tend to deal in symbols — they pay sur- 
prisingly little attention to their instructional core. Teachers, as a 
group, receive little supervision over their instruction. ^ 

What is more, some have argued, it is not in the schools 1 best in- 
terests to actually look very clearly at their instructional core if that 
core is weak. And it is widely recognized, that the actual technical core 
of teaching is quite weak. For example, educators have few really carefully 
researched and developed methods to deal with the challenge arising from an - 
influx of many non-English speaking pupils. Indeed, there are many dif- 
ferent viewpoints both about the advisability and techniques of bilingual 
education. In saying this, we do not suggest that educators are uncaring or 
that a little thought and research has not taken place with regard to this 
issue. Instead, we suggest that bilingual education, as an example, is a 
value-laden issue and a very complex instructional problem. Whatever the 
reasons^he fact is that schools have a very weak technical core. If 
schools look very critically at that core, and expose it to p.ublic scrutiny, 
the schools may begin to lose their institutionalized status. Some things 
are best left unexamined! In summary, this institutionalized characteristic 
of the schools lessens the schools' incentive to link testing, evaluation 
and instruction. / 

Goal Differences . The public schools have long -suffered from lack of 
agreement over the goals the schools should pursue. Readers with a sense of 

i 

S29 



- 225 - 



educational history can remember the movements from humanistic education to 
vocational education to basic skills. What is more, the goals of one seg- 
ment of society will often differ from those of other social segments. 
While there is general agreement on, say, the basic skills, there is consi- 
derable disagreement over priorities and over the importance and the best 
strategies for pursuing sudh other schooling functions as racial integration 
or developing critical thinking skills. Thus, schools are constantjy subject 
to a variety of external pressures to conform to the decision of one group 
or another. Given an open system orientation, it is important that they 
attend to^their clients 1 decision, but, as was pointed out earlier, these 
constantly shifting priorities make it difficult for districts to mount 4 
sustained, long term and carefully researched efforts to implement a given 
goal. Thus, school districts tend to sort of muddle through, bouncing back 
and forth between various changing demands that are placed on them. Insti- 
tutions in this kind of situation tend to disassert the need for a carefully 
designed management information and planning system — it is simply not 
worth the time and limited reserves it consumes. 

Permeable Boundaries . This feature has been described' in some detail 
in an earlier section of this chapter so we will not elaborate further in~ 
describing it. We- should repeat, "however, that this permeability makes it 
difficult for school districts to protect themselves from external activi- 
ties th$t both change the basic direction of the institution or direct a 
good deal of the district's attention and reserves away from the central 
instructional program. Many school districts, for example, have had to deal 
with demands placed on them from the external environment for which they 
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have been unprepared and which have demanded a good deal of their time. 
Examples are school programs to handle: desegregation, educating handi- 
capped pupils, large immigration of Latino or Asian pupils. We do not 
question the merits of such programs, only that they are so large in scope 
and complex in execution that they demand an enormous amou/it of time and 
resources and they force the schools to spread their planning and develop- 
ment capacities very thin. Districts so occupied with external demands are 
not likely to have the energy and resources to attend to their central 
instructional core, especially if the institutionalized character of the 
organization reduces the immediate need for them to do so. 

Loose Coupling . A currently popular term, used to describe the 
schools, loose coupling refers to the characteristic of some organizations 
to have loose couplings, or connections, among the organization's various 
parts or subcenters. Assume that an- order is given, by an organization's 
management, that a certain procedure or change is to be implemented. In a 
tightly coupled organization one will expect that the order or change will 
be carried out within a reasonable period of time. In a loosely coupled 
organization, the order or change may never be carried out or done so in a. 
sporadic or idiosyncratic manner. 

No organizations are completely tightly or loosely coupled. In 
schools, for example, if the school board orders that the beginning and 
ending hours for the district schools. be changed for the next school year, 
one can be almost certain that the order will be carried out. In that 
instance, the organization is tightly coupled. In the case of the 
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instructional program, however, if the school board orders that teachers 
institute a particular instructional approach or t>ase their instruction on 
the results of certain tests, one cannot be at all sure that after a reason- 
able time the order will be carried out. The school's instructional program 
is often loosely coupled from the organizational management system. 

There are, of course, reasons for this, the main one being that the 
organization's main technical core unit, the classroom, is operated quite 
independently by teachers. District managers have neither the resources, 
inclination nor agreed upon technical superiority to substitute their judg- 
ment for the teachers 6 over instructional matters. What is more, the vari- 
ous district subunits, especially in large districts, tend to operate in 
considerable isolation; they are themselves loosely coupled from one an- 
other. This loose coupling makes it unlikely that the testing, evaluation 
and instructional processes can be easily or naturally linked. 

Assumption 5. Developing and implementing a subsystem for linking a 
di strict's testing and evaluation together with instructional program 
(t /e/i) can be thought of as an educational innovation. 

As was pointed out in previous sections, it is not natural for school 
districts to ISn* testing, evaluation and instruction together. There are 
many institutional, organizational and historical factors that mitigate 
against such linkage. If such a linkage is going to be forged it will take 
some positive, direct and sustained planning and action. For some organi- 
zations, e.g., a manufacturing plant or an engineering firm, the use of an 
information management system that consists of the regular monitoring of 
critical technical core processes to determine whether they are being 
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completed as intended and whether or not they are having the intended effect 
may seem to be a standard operating procedure. Presently, in public school 
districts, however, such a process can be considered an innovation or 
change. 

During the past twenty years or so a great deal of research has been 
completed on the educational change process. While thte resultant research 
literature is too extensive to review here, there are a few generalizations 
that seem to emerge from that literature that would apply to districts want- 
ing to develop a t/e/i linking subsystem. The few generalizations that we 
feel apply are: the importance of ownership, the incremental nature of com^ 
prehensive change, the process of mittual adaptation and the role of incen- 
tives. \ 

The Importance o^ Ownership . As th^ name implies, ownership simply 
means that those who will be affected by an innovation have some sense that 
the innovation belongs to them. It conveys a feeding that those affected 
have at least been consulted about the innovation,, or better still, that 
they have played an important role in determining why, how and when the 
innovation will be introduced, implemented and adopted. This ownership can 
be developed in a number of ways, by having those who will be affected, or • 
some group representing them: be involved in developing the innovation; be 
participants in deliberations and discussions on whether or not to initially 
implement the innovation; be involved in judging whether or not the inno- 
vation is working as intended; and being involved in final adoption deci- 
sions. 
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Consider the differences between a school district that decides at the 
top administrative level to conduct, as an analytical step in building a 
t/e/i linking subsystem, an evaluation study to see if teachers are uni- 
formly spending enough time on mathematics and if they are using what the 
district considers "appropriate" teaching methods. Assume further that the 
decision is made quite unilaterally and the teachers are simply informed 
that this is an important study and that the data will not be used for per- 
sonnel purposes. One can pretty well predict what will happen. The teach- 
ers will be suspicious of the district's motives, they will resent the 
intrusion into their classroom, the study will become the target of teacher 
resistance, and this will likely minimize the accuracy and therefore the 
usefulness of the data gathered. 

Suppose instead, that teachers from the beginning were told that the 
district wants to work with them in improving instruction and that the re- 
sponsibility for whatever inadequacies there might be in*the instructional 
program belong to everyone including teachers, principals, % and central 
administration. Suppose further, that it is proposed that a joint committee 
be formed that consists of teachers, principals and relevant central admini- 
strators to explore the possibilities of conducting a study to determine 
current practice. And, vf such a study were deemed -Valuable, that the 
study's design, implementation and analysis would be directed, though not 
necessarily conducted, by this joint steering committee. Problems would 
still exist but one can predict that because the teachers, who are the ones 
to be studied, are involved there will likely be less resistance, the study 
will be more realistic in its design, execution and analysis, arid therefore 
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more useful to everyone. One risk of this approach is that the teachers may 
decide that the study should not be conducted because of reasons they con- 
sider important. If such is the case then the study should be either rede- 
signed so as to meet the teachers' concerns or forgotten. It is difficult 
to over-estimate the importance of ownership in designing and implementing 
innovations. 

* 

The Incremental Nature of Comprehensive Change . For many years, it was 
taken as an act of faith to describe the innovation process as a linear, 
logical process subject to standard pl-anning methods. Innovators were urged 
to state clearly their objectives, determine the resources needed, develop a 
detailed plan for .developing and implementing the innovation, and determine 
the appropriate evaluation mdthods and ways of feeding evaluation back so 
that the innovation could be properly adjusted. 

While this standard planning procedure makes sense logically and it 
looks good on graduate examination papers, it bears little relationship to 
what actually goes on when comprehensive innovations are introduced into 
complex organisations. There are likely many reasons for this, e.g., mar\y 
complex interactions are difficult to anticipate, not enough is known about 
the situation, or the innovation is not sufficiently developed so as to lend 
itself to such precise planning. We suggest, however, that perhaps the 
paramount reason is that educational innovations are being introduced into a 
human system that simply does not respond to such planning. Instead of 
being logical, linear and predictable, most innovations, when looked at in 
retrospect, are implemented on a non-rational, non-linear and incremental 
basis. And the patterns of implementation vary almost as much as the 
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variety of innovations themselves. This isn't to suggest that planning is 
to be completely avoided. Indeed, one should proceed with an innovation 
with a plan in mind. However, one should avoid being too wedded to a parti- 
cular strategy and one should be prepared, indeed expect, that the original 
plan will likely be changed considerably jfrorn time to time before the inno- 
vation is fully implemented. 

Mutual Adaptation . Related to the incremental planning concept de- 
scribed above, is the concept of mutual adaptation. Simply stated, this 
suggests that the chances of an innovation being implemented improve consi- 
derably if allowances are made for the innovation and the setting to mutu- 
ally adapt to each other. That is, the innovation of, for example, a new 
reading program, will likely undergo some adaptation as it is implemented in 
various sites.. It is unrealistic to believe that it can or should be imple- 
mented in exactly the same way in different settings. Similarly, it is. 
likely that the school settings will have to adjust themselves, though in 
different ways, in order to accomodate the innovation into their settings. 
These adjustments may be in the form of changing personnel or schedules or 
administrative procedures. It is not likely that any significant innovation 
can be introduced without some organizational adjustment. 

The Role of Incentives . For every innovation there are those indivi- 
duals and groups who expect to benefit from the innovation and those who 
will either receive 4 no benefit or will be disadvantaged by the innovation. 
For those advantaged, there are said/to be incentives. For those disadvan- 
taged, the innovation carries disincentives. It is not unlikely for those 
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who are disadvantaged to offer resistance to the innovation. Nor - is it 
unreasonable for them to be less than enthusiastic about an innovation, 
especially if they perceive the innovation as using them- to benefit someone 
else. 

The implication of this is that one should make every effort to design 
and implement innovations in suck a way that everyone benefits, and that the 
benefits are clearly spelled out to those who are participating. In our own 
^ work over the years we have come to view this as a cardinal rule of success- 
ful innovation. People will more likely cooperate with those innovations 
that help them solve a problem that they have. Few people can resist inno<- 
vations that solve real, pressing problems. Few people will assist innovat- 
ions that promise no personal payoff. 

In summary, those considering the development and implementation of a 
t/e/i linking subsystem should consider this as being so atypical of most 
school district functioning as to regard it as an innovation. As such the 
innovation should be planned and implemented in such a way that those who 
will be, affected by it will feel ownership for the plan, the plan' will be 
sufficiently flexible and sequenced that it can be changed to meet changing 
loc^j conditions and circumstances, he innovation will easily allow for 
mutual adaptation and the innovation will be so .designed and explained that 
all who will be involved in the plan will see in it ways to solve real and 
pressing problems. 
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CHAPTER 6 



School District Possibilities 

in which drstrict prsonnel, who are enroute to 
their own instructional renewal, examine their 
, own situations in terms of impediments as well 
as facilitating conditions • . • 
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Chapter 6 



School District Possibilities 



Sections 



1. Improving instruction through the management of testing and 
evaluation activities: an action planning perspective 

a) Introduction 

v b) Definition of action planning 

c) Attitudes on which district action planning rest 

d) Conditions impeding or facilitating action planning 



2. Alternative action plans: discussion and outlines 

a) Test-Focused *^ 
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c) School Planning-Focused 
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3. Tips and Techniques: 

a ) Tests : Cri teri on-referenced; Norm-referenced 
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4. - Initiating action planning: Discussion and Worksheets 
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Section 1 



Improving Instruction Through the Management of Testing 
and Evaluation Activities: An Action Planning Perspective 

241 
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Introduction 

Earlier chapters in this book deal with various aspects of the testing/ 
evaluation/instruction linkage. 

Chapter 2 describes a district in a crisis situation erupting from the 
public's reaction to publication of test scores. A number of issues are 
explored for example, the accuracy of the press' interpretation of the 
scores, the extent to which the scores serve as legitimate surrogates for 
student learning and as outcomes of the district's instructional effective- 
ness. More importantly, the scenario implies that the specific crisis is an 
instance "of a more chronic problem — one which is generic to many districts 
— that is, -the . lack of a central district level organizational structure 
which routinely and effectively feeds back the evaluative data periodically 
collected to assess student achievement into the instructional system which 
impacts student achievement. 

In Chapter 3, single factor solution strategies, often proposed by^the 
public and even by some educators, as responses to the immediate problem are 
presented, discussed and rejected. We reject them because we believe that 
taken individually, each points towards a dead-end direction. That is, each 
of them may be an important element in an integrated data-based instruc- 
tional management system. Singly, however, they are energy consuming diver- 
sions not making much of a contribution to educational environments which 
enhance student learning. 

In Chapter 4, six districts who recognized the interaction between 
assessment and action, between testing and instruction, tell how they devel- 
' oped their own data-based management systems. One district's system looks 
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very different from another district's/ system*. In all cases, however, im- 
proved student learning was the aim of. the system. . The teacher was recog- 
nized as the key to the delivery of effective instruction. The role of the 
school as the setting within which teachers work was acknowledged. District 
responsibility for supporting the principal and the teacher with relevant 
information and with data-informed policies was asserted. 

In Chapter 5, the generic difficulties experienced by school districts 
when they developed data-based instructional management systems are des- 
cribed. Many school districts these days exist in turbulent social, politi- 
cal and economic environments. Their budgets may decrease while their stu- 
dent needs increase. Their population may shift within the district causing 
some schQols to close, others to go on^double session. The .first language 
of many students may not be English. Student stress may be acted out in 
terms of drug use, vandalism, absenteeism, violence. Such turmoil and the 
rapidity with which non-educational problems seem to accumulate may engender 
cynicism or despair on the part of school officials that they can educate 
students at all. For these* educators, district management is focused on 
instructional survival rather than instructional improvement. 

In our view, school districts' need'to respond to rapidly changing envi- 
ronmental conditions is fuel for the development of a central office system 
for collecting, analyzing, and using test or other evaluative data to en- 
hance student progress and factors that seemingly contribute to or impede 
that progress. ' 

In this Chapter, then, we' discuss specific ways in which districts who 
are interested in such a point of view can engage in action planning to' 
create such a management system for themselves. 
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Definition of Action Planning 



As we use it here, action, pi anning is not the pre-ordinate specification 
of goals and objectives accompanied by a blueprint of activities and by a 
time/task line, although it might we"!! have some of these characteristics. 
Action planning is not , the arm-chair product of a single administrator's 
conception of a data-based instructional management system, although key 
individuals might well have decisive influence. On the other hand; neither 
is action planning a form of "muddling through" or "making it up as we go 
along" although at particular times and for particular individuals, it may 

well feel that way. 

Action planning, as we infer it from the "heroic districts" who have 
described their efforts in Chapter 4, requires district leaders with the 
vision to so shape action and events that they cumulate into new struc- 
tures. For example, in Los Alamitos, the unacceptability of the criterion- 
referenced tests first purchased by the district did not cause the abandon- 
ment of the data-based instructional plan, but rather refocused action on 
developing a set of district objectives. In San Juan, the difficulties with 
title I evaluations and program planning led not to abandonment of the idea 
of planning and evalution but to greater receptivity to the school site 
planning orientation of the California Early Childhood Education Program. 
In these two districts, as well as in the others in Chapter 4, individuals 
did not know at the outset of their endeavors precisely what their data 
mangement system would look like. Actions led to plans. These plans, in 
turn, gave rise to anticipated as well as unanticipated ideas and events 
which caused adjustments in subsequent actions and plans. 
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Action planning, therefore, can be characterized as simultaneously reac- 
tive and proactive, simultaneously spontaneous and controlled, combining or 
alternating both the doing and the reflecting- Action planning is a process 
which enables a district staff to accommodate to the complexities of .the 

external environments with which they are in constant interaction. It seems 

j 

consonant with the loose coupling of a school district's internal subsystems 
characterizing most school districts today. 

A ttitudes on Which District Action Planning Rest 
The professional attitudes which often accompany district action 
planning to create data-based instructional management systems can be 
divided into four categories. i 

Attitudes towards data. District personnel must believe in whole or 
in part — that 

° data from tests and from other evaluative devices such as surveys, 
observations, demographic statistics, are tools for describing the 
current status of students, for analyzing that status and for 
deciding how to upgrade it 

° data from tests and other evaluative devices can be collected, ana- 
lyzed and disseminated so as to be of use to a variety of audiences 
— e.g., teachers, principals, staff developers, curriculum speciali- 
sts, administrators — for a variety of purposes — e.g., clinical, 

programmatic, managerial, policy 

.// 

° the management of data from tests and other evaluative devices must 
be integrated with the management of 'classroom instruction, so 'that 
the two processes are com^tible and interact with one another 



Attitudes towards instruction. District personnel must belive ~ in 
whole or in part -- that 

° instruction can affect student learning in ways that can be measured 
by tests or other evaluative devices, 

° teachers can manage the elements of instruction — e.g., materials, 
time on task, teaching strategy — so as to affect student perfor- 
mance on tests 
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° teachers who deliver instruction to students are themselves part of 
and responsive to the school and the district context which influ- 
ences how and wha^ they teach 

° the district central office has an important role ~ facilitative or 
directive — in the instructional process as it occurs in their 
schools; the district has overall responsibility for the learning of 
the students who attend its schools 

Attitudes towards management . District personnel must believe — in 

whole or in part — that 

° strong central office management — of internal operations and of 
relationships to schools — can upgrade teaching and therefore effect 
student learning 

° central office management of tests, evaluations, curriculum, staff 
development, materials, can be integrated and coordinated through 
informal as well as formal mechanisms 

° there is no single "right way" to develop a data-based instructional 
management system that can be imported from elsewhere and implanted 
within the district 

Attitudes towards change. District personnel must believe — in whole 
or in part — that 

° the development of a data-based instructional management system may 
require changes in behaviors or in attitudes on the part of adminis- 
trators, principals, teachers, students 

° change in school districts is possible. It requires careful atten- 
tion to the factors and forces which have "frozen" the organization 
into its present configuration as well as understanding Gf the 
"unfreezing" and "refreezing" processes essential to organizational 
change and renewal 

° it is likely that change in school districts will be incremental 
rather than radical, gradual rather than sudden, uneven rather than 
orderly " 

° change may produce failures as well as successes 

° change may be threatening to some individuals or groups, energizing 
to others 
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Conditions Impeding or Facilitating Ac t ion Planning 



There are many conditions. within school district organizations which 

might impede the action planning needed to create data-based instructional 

management systems. We can list these briefly because they have already 

been discussed elsewhere. 

° rapid changes make it difficult to undertake a coordinated long term 
effort to create a data-based instructional management system 

° boundary permeability requires school districts to attend to or com- 
ply with legislative and administrative regulations whether or not 
these actually contribute to local educational improvement 

0 weak technical core for education . The fact that there is a very 
small number of research validated cause and effect relationships 
general izable to all teaching-1 earning situations encourages teacher, 
principal and administrator reliance on their own working knoweldge 
and insistance on their right to do so. 

° loose administrative coupling. Policy and administrative decisions 
taken at the board and central office levels may not filter down 
through school and classroom layers to directly affect students. In 
part, this may be due to imperfect communication devices, in part to 
differing role-related perspectives about what is important to do, in 
part to traditions within, the American public education system. 

There are, however, many conditions which facilitate action planning to 

link testing and evaluation with instruction. These can be devided into 

conditions external to the district and those internal to it. They will be 

presented here in the form of checklists. 
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External Conditions Favorable to Developing a 
Data-based Instructional Improvement System 

J 

□ stable student population with few sudden suifts in size or 
distribution 

D Community support for public education 

□ Community agreement on desired educational outcomes for their 
chil dren 

Q Community agreement on instructional methods 

□ Community acceptance of test scores as indicators of student learning 

t . > 

Q Community willingness to support district policies 
Q Parent willingness to support instructional change 

□ Media encouragement and willingness to learn about test score/school 
practices 

D Availability of external funds from governments, foundations, 
businesses, etc. 

□ Availability of consultant assistance from universities, professional 
organizations, county organizations, other districts 

□ Availability of research and development reports, journals, etc. 
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Internal Conditions Favorable to Developing a 
Data-based Instructional Improvement System 



C3 Strong "idea champions" in positions of power or influence 

Q Stable Core of supportive staff in central office and in schools 

D Individuals with technical and computer skills related to 

construction, selection, analysis of test or other evaluative data 

CI Individuals with substantive and technical skills related to 
curricular development 

CI Individuals with management skills related to team building, 
consensus development, communications,, etc. 

D Teachers interested in instructional improvement willing to make 
changes in their own methods, materials, classroom management, etc. 

Q Financial resources within the district to pay for system development 
and maintenance 

r 

Availability of computer services 
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Section 2 

Alternative Action Plans 
Discussion and Outlines 
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Discussion 



Each of the four plans presented .in outline form here represents a 



conglomerate of actions undertaken by districts in pursuit of their intended 



purpose. 



As is evident from the stories presented in Chapter 4, not all dis- 
tricts start from a clearly defined, explicit purpose. Some districts act 
their way into new kinds of thinking, while other districts may think their 
way into new kinds of acting. However, it is usually possible to infer from- 
the statements of key individuals, or from an examination of the activities, 
whether district intentions are narrowly defined in terms of raising student 
test scores or more broadly defined in terms of altering district management 
processes. 

Often, however, district intentions evolve over time. If impetus for 
change comes, as we see in our Chapter 2 Scenario, from public outcry around 
test scores, the immediate response of the district may be test-focused. 
However, as individuals in the central office, schools and classrooms think 
about strengthening the interaction between what teachers do, what students . 
learn, and the demonstration of learning on tests, district actions mey move 
in new directions. This is the essence of action planning as we have 
defined it. 

The four outlined plans are not exhaustive of all possible district 
foci, but they do seem characteristic of the range^ of school district action 
strategies. • 

The key difference among the plans is in the pivot point around which 
changes occur. "Possible payoffs" may appear similar among plans but the 
probability of their being achieved does differ from plan to plan. In Plan 
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A, for example, gains in student achievement test scores are highly likely 
in the short term whether or not important changes in instruction take 
place. By contrast, in Plan B, curriculum-based instructional changes may 
occur well in advance of any rise in student achievement scores. In Plan C, 
school site planning may well have beneficial consequences on school - 
climate, teacher morale and student learning but these consequences may or. 
may not be demonstrated by test score increases. The same* is true, at the 
district level, for Plan D. 

The bare-bones "action sequence" described in each plan are not 
prescriptive. They only indicate some of the steps that many districts have 
taken in pursuit of their explicit or implied purposes. Specific "how-to" 
suggestions which flesh out each of the plans are included in Section 3's 
Tips and Techniques. 

Our discussion of "district responsibility," likewise, is suggestive of 
methods in use by districts. Each district is unique in terms of its 
management style, its past history, its current situation; so each district 
must work out its own arrangements relating to directing, facilitating, 
consulting with, informing, supervising, supporting various groups. 

Finally, the "advantages" and "difficulties" are summary compilations 
of experiences which may be helpful. 
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Plan A: Test-Focused 



Explicit Purpose - to raise test scores for all students or particular subgroup(s) 
(e.g., language, SES, grade level, school) in all or a subset of 
tested skills (e.g., reading, math, language arts) 



1. Pivot point 



2. Possible 
payoffs 



4. District 

responsibilities 



currently administered test(s) (e.g., a norm-referenced test such 
as the CAT, MAT, CTBS, Iowa Test of Basic Skills; a locally devel- 
oped or purchased criterion-referenced test battery; State assess- 
ment tests; district or State proficiency tests) 



2.0 
2.1 



2.2 



2.3 



3. Action sequence - 3.0 

3.1 
3.2 



3.5 



Improved test scores 

Capability for tracking progress of individuals, subgroups, or 
total population over time; tracking of teachers, grades, 
schools by student scores. 

Provision of information to parents, board, media about stu- 
dent achievement as demonstrated by test scores 
Improved instruction 

Identify test or subtest(s) on which scores are to be raised 
Describe target students vfiose scores are to be raised 
Analyze knowledge or skills assessed by test 
Analyze past/current scores of students 
Act: provide students with practice in test-taking skills, 
and/or provide students with practice on items similar to 
those on the test, and/or provide students, prior to test, 
with instruction and materials targeted to knowledge and 
skills tested 

Analyze test score results, disseminate and plan for addi- 
tional action 

Distript administrators assume responsibility for raising stu- 
dent scores. With appropriate input from curriculum and test- 
ing specialists, from principals and teachers, they identify 
the test and the students. Analysis of the tested knowledge 
and skills can be done in the district office or by comiittees 
of teachers. Analysis of past and current scores might be " 
handled by a specialist. Development of action strategies 
(and the supporting materials or procedures to implement them) 
can "be handled either centrally, or by school, or by teacher 
committee. 



Advantages 



.0 Short term targeted practice for students on tested skills is 

likely to raise their scores expensively, withouf 

much system change. 
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Emphasis on raising, "test scores may diminish attention to 
non-tested learnings. 

Emphasis on raising test scores may encourage cheating; 
cause unproductive anxiety over test performance in teachers 
and children; cause excessive reliance on single indicator of 
student abilities. 
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Plan B: Curriculum/Instruction-Focused 



Explicit Purpose 



to improve classroom instruction for. all students or particular 
subgroup(s) (e.g., by language, SES, grade level, school), for 
all or a subset of specified objectives 



L. Pivot point 

I. Possible 
payoffs 



3. Action sequence 



4. District 

responsibilities 



5. Advantages 



- current teaching practices and materials relevant to specified 
objectives 

- 2.0 Improved instruction on particular objectives 

2.1 Improved student achievement scores on currently administered 
test(s) 

2.2 Upgraded teacher skills 

2.3 Improved classroom management 

2.4 Improved classroom climate 

2.5 Improved support system for teachers 

2.6 Improved supervision system for teachers 

-^.0 Describe target student group(s) 

3.1 Describe desired student outcomes, e.g., in reading, math, 
language arts 

3.2 Select or develop tests, evaluative outcome measures (e.g., 
CRT's, text-tests, teacher tests, surveys, unobtrusive mea- 
sures such as rates of absenteeism, vandalism, etc.) 

3.3 Analyze materials /instruction in relation to desired student 
outcomes 

3.4 Describe base line 

3.5 Act: coordinate texts: tests: teaching, and/or modify teaching 
through training/supervision, and/or provide supports to 
teachers: aides, pe....sts, materials, etc. 

3.7 Evaluate using outcome measures and disseminate results 

3.8 Take additional action 

District administrators assume responsibility for improving 
instruction. With appropriate input from curriculum, testing, 
evalution, staff development specialists, principals and 
teachers, they identify curricular objectives, present 
instruction (teaching methods/materials) as well as the 
changes in instruction vsfiich are called for. Central office 
staff coordinate their own operations in areas such as curric- 
ulum, testing, supervision, staff development in order to mon- 
itor, train, support, and assist teachers. 

- 5.0 Attention is focused on Improving those classroom instruc- 

tional practices including teaching methods, time-on-task, 
materials, etc., that are relevant to particular objectives, 
particular groups of students. ^ 
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6.0 Test scores for norm-referenced tests may remain low if such 
tests are mismatched with district curricular objectives or 
text materials. 

6.1 Instructional focus requires teacher support and willingness 
to adopt new methods, materials, etc. 

6.2 District-level coordination of support and supervisory 
relationships with principals and teachers may be difficult. 
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Plan C: School -^Focused 



Explicit Purpose - to improve school site' instructional' planning and assessment 



1. Pivot point 

2. Possibile 
payoffs 



3. Action sequence - 



- current school management 



2.0 Improved schqol management 

2.1 Involvement of principal, teachers, community in teams to 
develop school -wide problem identification, problem 
resolution, resource allocation 
School programming adapted to local conmunity needs 
Improved student achievement scores on currently administered 
tests 

Improved 'student knowledge, skills, attitudes on other 
measures 

Improved support system for teachers 
Improved supervision system for teachers 

Describe target schools 
Do needs assessments/problem surveys 
Identify higfa priority problems 

Analyze existing data about school -level performance on high 
priority areas or problems 

Describe desired outcomes: student, class-level, school-wide 
Select or develop tests or other evaluative outcome measures 
Identify and choose among alternative actions 
Act 

Evaluate using outcome measures 
Analyze and disseminate results 



4. District 

responsibilities 



5. Advantages 



2.2 
2.3 

2.4 

2.5 
2.6 

3.0 
3.1 
3.2 
3.3 

3.4 
3.5 
3.6 
3.7 
3.8 
3.9 

3.10 Take additional action 

District administrators assume responsibility for improving 
school site planning and assessment. With appropriate input 
from community, principals, teachers, parents, district faci- 
litates the organization of school site teams, provides them 
with the needed training, technical support, resources to do 
their own problem identification, action planning, implementa- 
tion and assessment. 

- 5.0 School effectiveness may be increased through community sup- 
port, regular planning and assessment procedures, ownership of 
solutions by principal and teachers, etc. 

5.1 Schools can develop solutions responsive to their own 
situation. 

5.2 Process may lead to increased morale, feelings of efficacy, 
school spirit, etc. 
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6. Difficulties - 6.0 Student test score increases iray or may not be a high priority 

school goal ; test scores may or my not be affected — short 
term or long term ~ by school site plans. 
Leadership, composition, productivity of school planning 
oroups rrcy vary greatly. 
6.5: Support, technical assistance, resources available to schools 
ioay be costly compared to observable benefits. 

6.3 Gains may be long term rather than short term. 

6.4 Sustaining time-intensive efforts ray be difficult. 
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Plan D: District Management-Focused 



Explicit Purpose - to coordinate all district operations towards the enhancement of 
student learning 



1. Pivot point 

2. Possibile 
payoffs 



3. Action sequence - 3.0 

3.1 



4. District 

responsibilities 



5. Advantaces 



6. Difficulties 



current district operations (e.g., testing, evaluation, curriculum, 
instruction, materials and media, supervision, staff development 

2.0 Improved district management 

2.1 Improved accountability system for principals, teachers 

2.2 Coordination of testing, evaluation, curriculum, instruction, 
texts, etc. 

2.3 Upgraded teaching skills 

2.4 Upgraded principals management skills 

2.5 Improved public image 

2.6 Improved district efficiency 

2.7 Improved student achievement scores on existing tests 

Identify arid publicize district philosophy in terms of student 
learning, teaching .processes, management characteristics 
Examine current situation in one or more areas by analyzing 
existing data, collecting additional information from needs 
assessments, problem surveys, instructional analyses, test 
scores, etc. 

3.2 Develop outcome measures, other indicators of progress 

3.3 Develop action plan for policy, administrative, school and 
classroom levels 
Act 

Monitor progress using outcome measures and indicators, and 
reassess 

Take additional action 

District administrators assume responsibility for coordinating 
internal operations as well as school relationships in such a 
^way that student J earning is enhanced. District takes leader- 
ship in identifying the needs, problems, associated with improv 
ing student learning; in directing or facilitating coordinated 
action planning to meet them; in* assessing progress. 

- 5.0 Examination and redirection of district management will lead to 

more effective central office leadership, supervision, informa- 
tion collection and analysis, etc. 

- 6.0 Test scores may not be affected in the short term by more 

efficient district management. 



3.4 
3.5 

3.6 
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Section 3 



Tips and Techniques: Criterion-referenced and Norm-referenced Tests 
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Criterion-referenced Tests 



How To Decide If You Should Consider a. Criterion-referenced Testing Program 

(If most of your answers are unboxed, you probably don't want a CRT 
system. If most of your answers are in the boxes, you should 
assume that a CRT system may be useful for your district.) 

1. Do the tests that you now give — either district-wide or by 
teacher choice — provide the teacher with information that 
helps him/her tailor instruction to student needs? 



NO 



YES 



Do the tests that you now give — either district-wide or by 
teacher choice — provide the teacher with information that 
helps him/her communicate with parents, aides, other 
teachers, about student needs? 



NO 



YES 



3. 



Do the tests that you now give — either district-wide or by 
teacher choice — provide the principal, teacher or parent 
with information that helps him/her understand Individual 
student progress across grade levels? 



NO 



YES 



Do the tests that you now give — either district-wide or by 
teacher choice — provide the principal with -information 
about the learrvirig of students by class, or by grade level, 
or by some other relevant subgroup? 



NO 



YES 



Do you have, or want to develop, a minimum or standard set 
of objectives that all, or identified subsets of, students 
will achieve? 



NO 



YES 



6. -Do you have, or want to, develop, a cross-referenced system 
that makes tests, texts, teaching time consistent with a set 
of district-wide objectives? 



NO 



YES 



Do you have, or want to develop, a set of remediation 
techniques ~ e.g., materials, programs, teaching methods, 
-- for students who are non-masters of particular objectives 
or groups of objectives? 



NO 



YES 



8. Do you have, or want to develop, positive teacher attitudes 
and teacher skill in "teach-test-reteach" and "diagnostic/ 
prescriptive" teaching strategies? 



NO 



YES 



ERLC 
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Criterion-referenced Tests 



How To Develop a Criterion-referenced Testing System 

1. Recognize that the development of such a system takes time — many 
districts have spent more than five years in developing a system. 

2. Recognize that there are tradeoffs for any major decision, and that a 
district's best judgment combined with knowledge of the benefits and 
costs is the best available method for making the decision. 

3. Allonte roles and responsibilities for developing the system. 

4. Keep all relevant groups — Board, unions, press, teache; s, parents, 
etc, informed as appropriate. 

5. Decide on a start-up development strategy, either 1) one subject (e.g., 
math, reading, language arts) for all grades; 2) major subjects for one 
grade. 

6. Compile a district curriculum scope and sequence by subject matter and 
grade level, stated in terms of student outcomes, by any of the 
following methods: 

a. use available subject matter scope and sequence done by text 
publisher; 

b. borrow another district's scope and sequence aM adopt or modify; 

c. infer from teachers what is already being done by subject by grade, 
write it down and refine it in terms of student outcomes; 

d. hire outside consultants to create sequence; 

e. train teacher committees to write student outcomes by grade and 
subject area. 

7. Identify important end-of-semester learning objectives, and develop item 
clusters for each objective. 

8. Identify important en-route objectives and develop item clusters for 
each objective. i 

9. Have teachers/subject matter specialists view items or revise. 

10. Try out items with students and revise. 

11. Package items into test booklets. 
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12. Develop administration procedures, e.g., timing and conditions of 
testing, and orientation for teachers. 

13. Develop scoring, and feedback procedures, e.g. ,. turnaround time, format 
for reviewing student, subgroup, class, grade achievement. 

14. Develop remediation policies for non-mastery students at group, class, 
grade, school, district level, as appropriate. 

15. Develop procedure to identify and handle objectives and tasks that need 
revi sion. 
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•Criterion-referenced Tests 



How To Purc hase a Criterion-referenced Test System 

1. Locate the objectives that the test items have been written to measure. 

2. Determine whechar tne tested objectives master district objectives; if 
not, is the district willing to change instruction to match test s 
objectives? 

3. Determine whether written items match stated objectives, i.e., can you 
infer objectives frjpm items? 

4. Determine whether objectives and the related test items are clustered in 
the sequence in which your district teaches the objectives. 

5. Examine the items. (Is the level of vocabulary, sentence structure, 
format, appropriate -to the grade level and students you want tested?) 

6. Determine whether test administration instructions are clear. 

7. Determine whether the scoring of student answers to items is quick, 
easy, easy for teachers to interpret. 

8. Determine whether there are alternate items available for pre/post or 
i etest purposes. 




9. Ask teachers to examine specimen sets of items. (What problems d&s^hf) 
see? Can the problems be resolved?) 

10. Find out about the services provided by the test company.-- training, 
scoring, interpretation, development of new items, tailoring of tests to 

\ your needs, etc. ~ and their costs. 

11. Weigh costs of purchasing tests against costs of developing tests. . 
(Think about dollars for salaries and time, available personnel, level 
of commitment, tradeoffs with other activities, length of time till 
payoff, etc.) 
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Criterion-referenced Tests 



How To Construct Your Own Criterion-referenced Tests 

0 

1. Specify the objectives the district wants tested. .(These may be end- 
of-semester and/or en-route objectives.) 

2. Describe the specifications for the items related to each objective r- 
that is, their content, the vocabulary, the types of distractors (for 
multiple choice formats), the criteria used in grading (for essay or 
short answer formats), etc. 

3. Develop item formats and_ sample items that describe a) instructions to 
students; b) stimula limits; and c> response, limits. 

4. Write several items for each objective. 

5. Review items for accuracy of content and instructional ambiguities as 
well as their conformity to specifications. 

6. Mix up the items and ask teachers to match items to objectives, or to 
infer objective from item. Eliminate items that seem irrelevant- to 
objective. 

7. Ask students to answer all items pertaining to one objective. Review 
and revise items that seem to cause trouble/ paying particular attention 
to those items missed by students'who answered most items correctly. 

8. Remember that you want a group of items, all of which measure the same 
objective, that students who have mastered the objective will answer 
correctly, and that students who have not yet mastered the objective 
will answer incorrectly. Ideally, you would like some multiple-choice 
items so that student errors will provide you with clues what about the 
non-mastery student has misunderstood or missed. 



How To Determine Test Length 

1. Recognize that test length is a tradeoff between getting an accurate 
measure of stucents 1 .knowledge and ski 11 s_( the more items, the more 
reliable the test), and other uses for the time spent dri testing. 

2. Three or four items per objective seems to be practical and feasible for 
most situations. More items should be used when important decisions are 
to be based on student test performance. 

How To Determine Passing Scores 

1. Recognize that passing scores are arbitrary and that mistakes can be 
made. 
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Set what appear to be reasonable cut-off points, e.g., 3 out of 4 items 
80 percent correct. Examine whether those students who "clearly pass 
and those who "clearly fail" would fall into the same categories if mea 
sured with some other criterion such as teacher judgment, classroom or 
homework performance, etc. Readjust cut-off point or revise items that 
do not seem to discriminate between "clear passes and clear failures. 

Set up review procedures for those students who fall between those who 
"clearly pass" and those who "clearly fail." 

Consider what remedies will be made available to non-passing students. 
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Criterion-referenced Tests 



How To Support an On-going Criterion-referenced Testing System 

1. Recognize that the tests themselves, keyed to a consensus set of 
objectives, are only one part of improving instruction through 
management of testing and evalution. 

2. Key curricular objectives end the criterion-referenced tests to 
materials/media. 

3. Key curricular objectives and the criterion-referenced tests to 
instructional time and methods. 

4. Develop procedures to find and handle difficulties' with items, formats, 
test administration, feedback to interested audiences. 

5. Sensitize groups to ways in which test data analyses can inform policy 
making and administration in areas such as staff development, budgeting 
school planning, communication to public and media. 

6. Remain aware of possible undesirable side effects, among them 



a. 


teaching 


test items 


b. 


narrowing 1n£ 


ruction to what is tested 


c- 


negle t.>rtg hi 


achieving students 


d. 


devotviy undu 


e time to testing 


e. 


fragmenting 1 


/V .ruction into measurable pieces^ 



7. Duild in p« ? nor review of curricula- objectives and tests. 
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Norm-referenced Tests 



How To Decide If 'to y Should Consider a t Norm-referenced Testing Program 

If most of j, our answers are unboxed, you probably don't want an NRT 
program. Zf msx. of your answers are in the boxes, you should 
assume thafc an NRT system may be useful for your district.) 

1. Do the tests that you now give — either district-wide or by 
teacher c^nice provide the teacher with information that 
helps hiro/har tailor instruction to student needs? 



NO 



YES 



2. Do the te>.ts that you now give — either district-wide or by 
teacher choice — provide the teacher with information that 
helps him/ner cqmmu;*"'cate with parents, aides, other 
teachers, cvac 



student needs? 



NO 



YES 



3. Do the tests \lv& you now give — either di stricf-wi de or by 
teacher choio — orovide the principal , teacher or parent 
with information that helps him/her understand individual 
student progress across grade levels? 



NO 



YES 



4. Do the tests triat you now give — either district-wide or by 
tedchttr choice — provide the principal with information 
•f.fcotrv the learning of students by class, or by grade level, 
c? by some other relevant subgroup? 



NO 



YES 



5. Do you have, or want to develop, a minimum or standard set 
of objectives that all, or identified subsets of, students 
will achieve? 



NO 



YES 



6, Do you have, or want to develop, a cross-referenced system 
that makes tests, texts, teaching time consistent with a set 
of district-wide objectives? 



NO 



YES 



7. Do you have, or want, to develop, a set of remediation 

techniques — e.g., materials, programs, teaching methods, 
— for students who are non-masters of particular objectives 
or groups of objectives? 



NO 



YES 



8. Do you have, or want to develop, positive teacher attitudes 
and teacher skill in "teach-test-reteach" and "diagnostic/ 
prescriptive" teaching strategies? 



NO 



YES 
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9. Do you need a test that will compare district students' per- 
formance with others of similar background oh particular 
skill areas? 



NO 



YES 



10 Is a norm-referenced test required by law or regulation to 
evaluate federal or state programs? 



NO 



YES 



11. Do the Board, the public, the media, parents, colleges, 
important others, insist on comparative data on a 
nationally-known test? 



MO 



YES 



12. Can you locate a norm-referenced test which meets the mini- 
mum requirements of validity for your district's student 
population, curricular objectives? 



NO 



YES 



ERIC 
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Norm-referenced Tests 



How To Purchase a Norm-referenced Test System 

1. Choose a test that is matched as closely as possible to district 
curriculum objectives. To determine this: 

a. classify the test items from the test form according to district 
curricular objectives; 

b. tally the number of district objectives for which there are one, 
several, or man^ test items included the test; I 

c. tally the number of test items connected to those district 
objectives taught to students prior to testing. 



2. ~ Determine from the manual if the test is normed with students similar to 

those in your district at the time of year you plan to give the. test. 
(If not, consider how this would affect your interpretion of the scores, 
if you use this test.) 

3. Review the test for negative factors such as 

a. test bias in content, vocabulary, pictures, etc. 

b. confusing test administration requirements 

c. confusing test format/instructiorts to students 

d. higher tharv usual per-pupiV costs (20 cents per pupil is low, 
80 cents at- the high end) 

4. Find out what services and score analyses are available from publishers. 



5. Involve a variety of individuals in test selection procedures, e.g., 
curriculum and test specialists, administrators, teachers, parents, etc. 

i ' 

6. Review as many relevant tests as possible. Get names from Buro's Mental 
Measurements Yearbook , published by Gryphon Press, Highland Park, TJew 
Jersey. ~~ — 
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Norm-referenced Tests 



How To Interpret Norm-referenced Test Scores 

1. A student's percentile score includes the percentage of the norm group 
whose raw scores fall below the student's raw score. For example, 
performing at the 60th percentile means that a student's raw score was 
higher than 60 percent of the students in the norm group. Test pub- 
lishers should provide percentile scores for grades or schools as -they 
cannot be obtained by averaging individual percentiles. 



2. A student's standard score indicates how far above or below the norm 
ij^oup mean ot raw scores that student's raw score fell. 

f 

3. A student's stanine score also indicates how far a student's raw score 
deviates from the norm group mean of raw scores. The raw scores are 
divided into nine intervals containing a fixed percentage of the raw 
scores. For example, 4 percent of the raw scores fall in stanine 1 and 
in 9; 7 percent of the raw scores fall in stanine 2 and'in 8; 12 percent 
in stanine 3 and in 7; 17 percent in stanine 4 and in, 6; 20 percent 

in stanine 5. If a student has a stanine of 9, he/she has done better 
than 96 percent of normed students. 

i 

4. A student's grade or age equivalent score tells wtfer& his/her raw score 
falls with respect to the average performance of students at various 
grades or age levels. There are many methodological problems with these 
scores and many experts recommend they not be used. ■ 

l 

5. Consult with test publishers to determine most useful analyses for dis- 
trict purposes, e.g., by individual, class, grade, school; by student 
characteristics of instructional relevance, such as language, SES, 
length of time in school, etc.; by subtests. 
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Norm-referenced Tests 



How To Use Norm-referenced Test Scores 

To group or place students . Examine scores on particular content or 
subject areas and group students with similar skill profiles or group 
heterogeneously according to some instructional philosophy. 

2. To diagnose student needs. Examine subtest results to determine 1 ndi - 
vf dual pattern of achievement/deficiency. Look for surprises, that is, 
students who do better or worse than expected on all or some subtests. 

3. To evaluate instructional programs. Compare actual performance of 
students with expected performance. Examine reasons for correspondence 
discrepancy, e.g., is it in the test? the instructional program? or 
the match between test and instructional program? 

Or, compare students' gains from year to year to get gross estimates of 
actual performance versus expected performance ^over time. Examine 
reasons for correspondence or for discrepancy."^ 
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Tips and Techniques: Evaluations 

/ 
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Evaluations (List of topics) 



How to develop non-test evaluations of student achievement 
How to develop non-test evaluations of instructional programs 
How to regularize collection of non 7 test evaluative data 
How to disseminate findings from non-test evaluative data 

How to insert evaluative data into policy and administrative decision 

making * • 
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Tips and Techniques: Curriculum and Instruction 



ERIC 
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Curriculum and Instruction (List of topics) 



How to decide if you need a uniform set of curricular objectives 
How to develop curricular objectives by subject areas, by grade 



How to identify milestone objectives for which test items could be 
developed 

How to index curricular objectives to text and other materials 

How to assess instructional time/amount of practice students receive 
on curricular objectives 
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Tips and Techniques: School Site Planning and Assessment 
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School Site Planning and Assessment : (List of topics) 

How to decide on the desirability of school site planning and 
assessment 

How to initiate school site planning and assessment 

How to form, facilitate, train and support school site teams 

How to determine data needs of and data presentations to school 

teams 

How to assess whether school site teams are cost effective 

/ 



i 
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Tips and Techniques: District Management 



( 




- 274 - 



District Management: (List of topics) 



How to identify .v.- "Is for coordination among district operations such 
as testing, evalu< . ■ »n, curriculum, instruction, staff development 

How to select and i;,.>? ement appropriate coordinating mechanisms among 
district operations 



How to collect, analyze a < <;seminate data useful in policy 
derisions 

How collect, analyze a* 1 ..^■•Viat* data useful in school 
impr vvei.ient 

How to ■■;^f»ect, analyze a>*-, '"'isenrlnate data useful in classroom 
decisio ; ir* : ng 

How to oroide school -level supports, supervision and training so 
that jpnncipaU and teachers use test and evaluative data to improve 
instruction 



\ 
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Section 4 



Initiating Action Planning: Discussion and Worksfieets 
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Discussion 

The six worksheets here are intended as guides .or agendas for groups of 
people assembled by district administrators for the purpose of initiating 
planning to improve instruction through the management of testing and evalu- 
ation activities. 

It is anticipated that many of the conditions facilitating action 
planning are present in the district before this process is begun. For 
example: 

1) an external environment which is supportive of 
instructional improvement, and the use' of test scores 
and other evaluative data to report on student progress 

2) idea champions and supporters within the district 
who have the motivation, skills and power to move the 
district in this direction 

3) resources and time to consider alternative action plans 
and to manage the implementation of decisions resulting 
from that consideration 

Worksheet #1 examines current practices. Its intended outcome is based 
on understanding of constraints — those that are fixed and those that are 
alterable. 

Worksheet #2 explores satisfaction and dissatisfaction of various indi- 
viduals and groups with current practice. Its intended outcome is identifi- 
cation of agreed-upon troublesome areas or problems. p 

Worksheet #3 surfaces recent attempts or thoughts about change as we^ll 
as the supporting or opposing parties. Its intended outcome is arraying the 
reasons why changes have or have not come about. 

Worksheet #4 elicits ideas and suggestions for. action planning and iden- 
tifies where support or resistance is likely to come from. Its intended 
outcome is an informal identification of priorities. 
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Worksheet *5 is a force field analysis for high priority areas. Its 
intended outcome is d detailed description of factors influencing action 
planning. 

Worksheet #6 is a further listing of factors influencing action 
planning. Its intended outcome is the formation of an action plan. 
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1. WHAT IS THE DISTRICT NOW DOING IN THE AREAS OF TESTING, EVALUATION, CURRICULUM, INSTRUCTION? 



1. What tests is the district now using? For each of the tests, think about: 1) who/what influenced the selection of 
that test; 2) who is now interested in having the test given; 3) who now sees the test scores; 4) what use, if 
any, do various groups of people (e.g., parents, t ors, principals, central office, press) make of the scores; 
5) what would happen if the test were not given? 

i 
i 

2. What evaluations does the district now do? For each evaluation activity, think about; 1) why is the evaluation 
done?. 2) Who is interested in having the evaluation done? 3) What use, if any, do various groups of people 
(e.g., parents, teachers, principals^ central office, press) make of the findings? 4) What would happen if the 
evaluations were not done? 



3. What curriculum does the district now have? What are its characteristics (e.g., to what extent does it guide 
teachers 1 choice of content, skills; to what extent is 1t consistent with texts, tests; to what extent is it 
differentiated for subgroups of students)? What puld happen if the test were not given? 

4. What instructional policies does the district now have? To what extent are they implemented in the classroom? 
Who /What influenced the development of the policies? Who oversees th* ir implementation? What would happen if the 
instructional policies were changed? ■ ' 



ANALYSIS 



What is now being done that must remain in place? 
What changes could be made? 

What factors must be considered when making changes? 



2. HOW SATISFIED ARE VARIOUS GROUPS WITH VMT TT£ DISTRICT IS mi DOING IN TESTING, EVALUATION, CURRICULLM, INSTRUCTION 



TESTS 



la. What are areas of satisfaction 
with current tests? 



Who holds 
these views? 



lb. Viiat are areas of dissatisfaction 
with current tests? 



EVALUATION 



Who holds 
these vierfs? 



2a. What are areas of satisfaction 
with current evaluation activities? 



Who holds 
these views? 



2b, What are areas of dissatisfaction 
with current evaluation activities? 



CURRICULUM 



3a. What are areas of satisfaction 
with current curriculum activities? 



Who holds 
these views? 



What are areas of dissatisfaction 
with current curriculum activities? 



Who holds 
these vievs? 



Who holds 
these vierfs? 



INSTRUCTION 



4a. What are areas of satisfaction with 
current instructional activity? 



Who holds 
these viavs? 



4b. What are areas of dissatisfaction Who holds 
with current instructional activity? these views? 
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ANALYSIS 



What are areas of consensus and cfevelopiiBnt among various groups? 
What are major areas of satisfaction? Major areas of dissatisfaction? 
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3. WWT CHANGES HWE BEEN RECENTLY CONSIDERED OR NttDE IN TESTING, EVALIATIOM, CURRICULIM, INSTRUCTION? 



L in the area gf testing: (e.g., eliminating, replacing, adding tests; changing dates; changing analysis; changing dissemination, 
ot results, etc.) r - 



changes considered 



who suppported/why? ' 



who opposed/Wiy? 



what happened? 



2. in the area of evaluation: (e.g., eliminating, modifying, adding evaliations; changing questions, design, instruments, 
analysis, '.dissemination or results, etc.) 



change 1 considered 



who suppported/why? 



who opposed/why? 



3 in the area of curriculum: (e.g., eliminating, revising, redoing) 

changes considered who suppported/wty? .who opposed/wliy? 



4, in the area of instruction: 
changes considered . 



who suppported/why? 



who opposed/v^iy? 



what happened? 



vJiat happened? 



what happened? 



ANALYSIS 



What changes have been made and what were the reasons they were made? 
Who were internal and external "idea champions"? Supporters? 
How did they influetffce the changes? 
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4. IN THE AREAS OF TESTING, EVALUATION, CURRICULUM, INSTRUCTION 
WHAT WOULD VARIOUS GROUPS LIKE TO HAPPEN? 



The community /press 
would like 



The school Board 
would like 



Central Office staff 
wo^ld like 



e.g. 



Principals woulcflike 



Teachers would like 



Students would like 



Parents would like 



^erlc 



ANALYSIS 



What are areas of consensus and disagreement among various groups? 
Can desires be categorized: Easy/Difficult; Inexpensive/Expensive; 
What high priority areas for action planning can be identified? 



Long-term/Short-term? 
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5. FDR EACH HIGH PRIORITY /KLA LIST FACTORS FACILITATING/OPPOSING CHANGE AND THEIR STRENGTHS 



Strength 
High Med. Low. 



Factors Facilitating Change 



Factors Opposing Change 



CHANGE 



> Strength ' • 
Hign Med. Low 



ANALYSIS 



Can the factors facilitating change be increased; factors opposing change 
decreased? . • ' 

/Among all the priority areas which seems the mostL feasible plan to begin? 



V 



6.. FDR THIS HIGH PRIORITY AREA, SHOULD THE DISTRICT BEGIN ACTION PLANNING? 



1. State the explicit purpose for district action 
Who shares them? • t 

Who opposes them? . 

2. State the auxiliary payoffs desired ty the district 
Who* wants them? * < „ 

.Who opposes them? 

3. What are cost estimates for action? 

4. What needed resources, e.g., skilled personnel, 
materials, equipment, etc., are available? 

5. What is a reasonable timeline for action? 



v 



6. What should the roles. and responsibilities 
of the Board be,' if any? 

7. What should the roles and responsibilities 
of district administrators be, if any? 



00 
CO 



8. What should the roles and responsibilities 
of principals be, if any? 



9. What should the roles and responsiblities 
of teachers be, if any? 



10. What problems can be anticipated? 

11. How will outcomes of the actions be assessed? 



eric 



/WALYSIS 



Is there sufficient clarity of - purpose, support, resources 
role definition to rove on this priority? 
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